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I. INTRODUCTION 



This is Volume II of the report on the evaluation of the NSLP Application Verification Pilot 
Projects. It supplements Volume I, which presents the evaluation findings. Volume II has two 
objectives: (I) to provide a detailed description of the methods used to conduct the study, and 
(2) to present tabulations that supplement and extend the analyses reported in Volume I. 

To provide context for the discussions in subsequent chapters, it is useful to keep in the 
forefront the main features of the study design: 



• Comparison districts, which were carefully matched to each pilot district, provide a 
benchmark for measuring the impacts of the Up-Front Documentation and Graduated 
Verification Pilot Projects on the certification of students from eligible and ineligible 
families for free and reduced-price school meals, and on the participation of students 
in the NSLP. 

• Stratified, scientifically selected samples of all families (with the exception of those 
directly certified) in each school district provide the basis for drawing inferences 
about impacts. 

• Surveys of sample students’ families used carefully structured questionnaires to 
derive independent measures of household size and income, with which to assess the 
income eligibility of each student family selected for the study. 

• We used data provided by school districts on the meal price status of each selected 
student as of October 31, 2002, to measure certification for free and reduced-price 
meals. 

• We drew inferences about impacts of the pilot procedures by comparing the 
experiences of the sampled student fa mi lies in the pilot and comparison districts, and 
by using statistical models to control for the effects of family characteristics that both 
influence the experiences of interest and differ between pilot and comparison 
districts. 

• We obtained data on implementation of the pilot procedures and the costs of 
administering the NSLP application and verification processes through semistructured 
interviews with staff who conduct these activities in each district. 



In each district, we selected a stratified sample that was designed to achieve precision 
objectives for three groups: (1) students whose family income made them ineligible for free or 
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reduced-price benefits (issue of deterrence); (2) students whose family income made them 
eligible for benefits (issues of barriers); and (3) students who were certified for free and reduced- 
price meals (issues of accuracy). Survey data collection used a two-step process designed to 
determine accurately the income eligibility of each student’s family, while minimizing the 
burden on respondents and the costs of data collection. We used a part 1 interview administered 
primarily by telephone to make a rough estimate of income relative to poverty level. Households 
whose income was 400 percent or more of the federal poverty level according to this rough 
estimate were classified as having income above 185 percent of poverty. For households whose 
income based on the part 1 interview was less than 400 percent of the federal poverty level or 
who did not complete the part 1 interview, we administered a part 2 interview in person and 
gathered detailed information on income and household composition. 

The sampling and data collection schedule was designed so as to meet the objective 
providing FNS with preliminary tabulations of the data by the end of February 2003. To allow 
time for processing the data and preparation of the tabulations, this end point required that data 
collection be conducted between October 15, 2002, and December 15, 2002. On the other hand, 
key lists necessary for constructing sample frames became available only when school opened in 
August or September 2002. These constraints shaped our approaches to sampling and data 
collection in important ways, as explained in Chapter III and Chapter IV of this second volume 
of the report. 
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II. COMPARISON SITE SELECTION AND RECRUITMENT 



A critical step in the success of the NSLP Application/Verification Pilot Projects Evaluation 
involved selecting comparison districts that were very similar to each of the pilot districts 
originally included in the evaluation. The remainder of this chapter describes the general 
strategy used for selecting comparison districts, summarizes the details of the comparison district 
selection process for each of the pilot districts, and describes the success of efforts to recruit the 
comparison districts selected as the best matches for each pilot. 

A. GENERAL PROCESS FOR SELECTING COMPARISON DISTRICTS 

In the Request for Proposal published by FNS in January 2002, the study included 14 pilot 
sites for which comparison districts had to be selected, and our general design called for 
selecting one comparison district for each pilot district. The strategy for selecting pilot districts 
consisted of three steps: (1) restricting the choice set of potential comparison districts, (2) 

ranking these potential comparison districts in terms of the similarity of their measurable 
characteristics (at baseline) to those of the pilot districts using a quantitative similarity index 
(QSI), and (3) refining the ranking of potential comparison districts based on conversations with 
individuals familiar with the area in which the potential comparison and pilot districts are 
located. Once we selected the best candidates for comparison districts, we contacted them to 
recruit them to participate in the evaluation. 

One of the original pilot districts — St. Mary’s School in Paterson, NY — was dropped from 
the evaluation prior to the selection of a comparison district because of difficulties it experienced 
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in implementing the Graduated Verification model.' In particular, because of limited staff 
resources at St. Mary’s, they were unable to complete all of the second round verifications in the 
2001-2002 school year. Furthermore, the district did not have the data systems necessary to 
implement the requirement that students whose benefits were reduced or terminated in the 
previous school year submit income documentation with their certification application in the 
current school year. Thus, FNS decided that the level of pilot implementation in St. Mary’s was 
so low that they would not be able to provide a meaningful test of the basic model, and the 
school was dropped. 

1. Restricting the Choice Set 

In order to limit the range of possible comparison districts, we first excluded districts that 
did not share a few key characteristics with the pilot district. The underlying assumption here 
was that matching pilot and comparison districts on these characteristics was so critical that it 
was not worth considering districts that did not match on these characteristics. Using data from 
the Common Core of Data (CCD) from the National Center for Education Statistics (NCES), we 
selected as potential comparison districts those that matched the pilot in the following ways: 

• They were from the same state; 

• They were public school districts, like the pilots 

• They served the same grade levels 



*A second pilot district — Morenci, AZ — was dropped because we could not identify an 
appropriate comparison district. This situation is discussed below. 

2 

St. Mary’s School District was the only private district among the original 14. Since St. 
Mary’s was dropped, we were left with only public districts as pilots in the evaluation. 

-5 

In other words, we wanted to match to pilot districts containing only high schools 
comparison districts with only high schools covering the same grades. However, we did allow 
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Originally, we had intended to restrict the choice set to potential comparison districts that 
matched pilot districts in terms of whether they were located in an urban, suburban, or rural 
location. In the course of restricting the choice set of potential comparison districts, however, we 
found that the urbanization classification of some pilot districts seemed to provide relatively little 
useful information, in that they appeared to be less similar to other districts in their urbanization 
category than to districts in other categories. As a result, restricting the choice set to only 
districts with the same level of urbanization would have excluded many potentially good 
comparison districts. 

In addition to the characteristics listed above, it would also have been desirable to have 
restricted the choice set to districts that matched the pilot in terms of whether they used direct 
certification and whether they included any provision 2 or 3 schools (none of the pilots included 
provision 2 or 3 schools). Unfortunately, no data were available for potential comparison 
districts that would have allowed us to exclude those districts that did not match the pilots along 
these dimensions. Instead, we decided to wait until we had selected a small number of good 
matches for each pilot district based on observable characteristics and determine whether the 
comparison and pilot districts also matched in terms of direct certification and provision 2/3 
status. 

The number of potential comparison districts in the choice sets of each pilot varied widely, 
depending on their state and the other relevant characteristics. The smallest choice set was for 



(continued) 

potential comparison districts to differ from pilot districts with respect to whether they served 
preschool students. 
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Oak Park/River Forest, a high school district in Illinois, for which there were 71 potential 
comparison districts. The largest choice set was 600, for Dunkirk, in New York. 

2. Creating the Quantitative Similarity Index 

The second step in the process of selecting a comparison district for each pilot was to create 
a quantitative similarity index (QSI). The purpose of this QSI was to measure how similar in 
measurable baseline characteristics each potential comparison district was to the pilot to which it 
was being matched. The QSI was calculated as a weighted sum of a set of QSI components. 
Each QSI component represented a dimension on which we wanted to match the comparison 
district to the pilot district. The weight assigned to each component reflected the relative 
importance assigned to each dimension in ensuring the similarity of the comparison and pilot 
districts. 

We measured the district characteristics used as QSI components using information for 1999 
and obtained from the CCD. The individual QSI components, grouped into categories, included: 



• District Size 

Number of schools in the district 
Average number of students per school"^ 

• Race/Ethnicity 

Percentage of students who were non-white^ 



"^Originally, we defined this component as the total number of students enrolled in the 
district. However, we felt that by including both the number of schools and average number of 
students per school we could better capture both the overall size of the district and the size of the 
individual schools in the district (which would be relevant if the district administered the NSEP 
in a decentralized fashion). 

^In cases where the pilot district had substantial proportions of more than one group of non- 
white students, more than one race/ethnicity category was used. Eor example, several pilot 
districts included substantial proportions of black students and Hispanic students, and in these 
cases we included both “percentage white” and “percentage black” or “percentage Hispanic” 
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Percentage of students who were limited English proficient^ 

• Proximity 

Number of miles between pilot and comparison district 

• Poverty 

Percentage of children ages 5 to 17 in district who were in households with 
income below 100 percent of the Federal Poverty Level in 1997 

• Certification Rates 

Percentage of students certified for free meals 

Percentage of students certified for free or reduced-price meals’ 

• Participation Rates ^ 

Participation rate among students certified for free meals 
Participation rate among students certified for reduced-price meals 
Participation rate among students not certified 

For each of these district characteristics, we calculated the value of the QSI component 
reflecting that characteristic using two steps. First, we measured the difference between the 
value of the characteristic for the potential comparison district and for the pilot district. Second, 



(continued) 

components. Also, in one state data on race/ethnicity was not available in the CCD, so we used 
Census data. 

^Information on limited English proficiency was not available for all states. In cases in 
which it was not available for a substantial number of districts, it was not used as a QSI 
component. 

•y 

Originally, we included separate components for the percentage of students certified for 
free meals and the percentage certified for reduced-price meals. However, we decided against 
including the percentage certified for reduced-price meals alone so that it would not contribute 
too strongly to the overall QSI (since a fair amount of variation in the percentage certified for 
reduced-price meals appeared to be somewhat random). 

o 

Information on NSLP participation was not available from FNS administrative data for all 
states. For districts in states without participation data, we dropped the participation components 
and re- weighted the remaining components so that the weight originally assigned to participation 
was given to the certification components. 
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we normali z ed this difference. The normalization process consisted of dividing the absolute 
comparison-pilot difference in the characteristic by the total range in the value of the 
characteristic across all districts. The advantage of this procedure is that if the pilot district has 
the maximum/minimum value on the characteristic, a comparison district with the 
minimum/maximum value will receive a relative difference value of 1.0, which represents a 100 
percent deviation from the pilot district. With this approach, the values of each QSI component 
can range from 0 to 1 and can be interpreted as the departure of the comparison district to the 
pilot district with respect to that characteristic. Formally, the QSI component value for district 
characteristic X was calculated as follows: 

QSI X = ^ ^ 

X — X 

max min 



Once the QSI component values for each district characteristic listed above had been 
calculated, we calculated the overall QSI value. To do this, we used the weighted average of the 
individual QSI components. Although we examined several different weighting schemes, the 
one we ultimately chose was based on our desire to match the pilot and comparison districts 
closely on their baseline certification and participation rates, since these would be key outcome 
measures in our analysis. We also wanted to match closely on the poverty rate in the district. 
For both theoretical and operational reasons, the proximity of the pilot and comparison districts 
was also important, though not quite as important as certification, participation, and poverty. 
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Thus, we settled on a weighting scheme that assigned weights to the QSI components as 



follows:^ 

• 25 percent of the overall weight to the three participation components 

• 25 percent of the overall weight to the two certification components 

• 25 percent of the overall weight to the poverty component 

• 15 percent of the overall weight to proximity 

• 5 percent of the overall weight to the race/ethnicity components 

• 5 percent of the overall weight to the two district size components 

After generating the value of the overall QSI for each potential comparison district in the 
choice set, we ranked these districts according to their QSI values. Districts with the lowest 
overall QSI scores were considered the best matches for the pilot district and were ranked 
highest. We then selected a smaller set of potential comparison districts with the highest 
rankings for further consideration. In most cases, we selected the top ten potential comparison 
districts for this purpose. In some cases, however, we determined that there were not ten districts 
that would make suitable matches for the pilot district and so we reduced the number of potential 
comparison districts selected for further consideration. 



^ Aside from the final weighting scheme, we tried six other weighting schemes. These 
alternative schemes gave different weights to the different components. These included one 
scheme that gave each of the components equal weight, another scheme that gave each category 
of components equal weight, and various schemes that were similar to the final weighting 
scheme in giving relatively large weights to certification, participation, and poverty, but varied as 
to the exact amount of weight placed on each. We settled on the weighting scheme presented 
here because it resulted in a ranking of potential comparison districts that had the greatest face 
validity. 
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3. Refining the Ranking of Potential Comparison Districts 

To obtain further information on the suitability of the top ranked potential comparison 
districts, we spoke with individuals familiar with both the pilot district and potential comparison 
districts. These individuals included state-level NSLP administrators along with staff at the pilot 
district. In a few cases we consulted with individuals at nearby colleges or universities whose 
areas of expertise made them knowledgeable about the local area.'° For some of the pilot 
districts, we held these conversations by phone. In other cases, we visited the pilot districts 
and/or potential comparison districts. 

In our conversations with these state and local staff, we typically presented them with a list 
of the top ranked potential comparison districts and asked them to assess the comparability of 
these districts with the pilot district. We asked specifically whether these districts matched the 
pilot districts in terms of their direct certification and Provision 2 or 3 status. We also asked 
whether they were similar to the pilot in terms of their administration of the school lunch 
program. For example, we asked whether the comparison and pilot districts matched in terms of 
their degree of centralization and their use of food service management companies. Finally, we 
asked the state/local staff to point out any important factors that might not have been apparent in 
the CCD data we used to calculate the QSI and rank potential comparison districts. 



*'*This strategy was employed in the comparison district selection process for the Morenci, 
Dunkirk, Dilworth-Glyndon-Felton, and three eastern Pennsylvania pilot districts. 

"in one case, for example, we were told that while the potential comparison districts might 
have matched the pilot in terms of district size, it was not clear that they matched in terms of the 
growth of the school district in recent years. In that case, the pilot district was located in an area 
that had been experiencing rapid population growth, while the areas in which some of the 
potential comparison districts were located were not growing rapidly. Since information on the 
rate of population growth was not one of our QSI components, we would not have known this 
without the input from the state and local staff. 
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Typically, the conversations with state and local staff resulted in the selection of a small 
number of districts that were clearly above the rest as the best matches for the pilot district in 
question. In a few cases, there was only one potential comparison district that emerged from 
these discussions as a suitable match. And in a couple of cases, none of the potential comparison 
districts identified by our initial QSI analysis were thought to be good matches, so additional 
work was required to find appropriate districts. Ultimately, however, we settled on one district 
that we thought to be the most promising candidate as a comparison district, along with one or 
more backup districts that we would turn to if the original district declined to participate in the 
evaluation. 

In this process of refining the ranking of potential comparison districts and targeting one or 
more for our recruitment efforts, we took two other factors into account. First, although free or 
reduced-price certification rates and the poverty rate were included in the original set of QSI 
components, we also came to the conclusion that the ratio of these two factors was important. In 
particular, in cases where the certification and poverty rates of potential comparison districts 
deviated somewhat from those of the pilot district, we would favor districts in which the ratio of 
the certification rate to the poverty rate was close to that of the pilot district. Second, as we 
proceeded with the selection and recruitment of potential comparison districts, we began to take 
into account the overall balance of characteristics between all pilot districts and all comparison 
districts. In other words, if the mean baseline certification rate of all comparison districts that 
had already agreed to participate in the evaluation was greater than the mean baseline 
certification of the pilot districts to which they were matched, this would influence our selection 
of subsequent comparison districts. In particular, this situation would lead us to favor a potential 
comparison district whose baseline certification rate was slightly below that of the pilot district 
over a potential comparison district whose baseline certification rate was slightly above that of 
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the pilot district. In this way, we could bring the overall certification rate among all comparison 
districts a bit closer to that of all pilot districts. 

4. Recruiting Comparison Districts 

Selection and recruitment of comparison districts was a joint activity carried out by the FNS 
project officer and senior MPR project staff (project director, principal investigator, and survey 
director). In the discussions with individual candidate districts, the FNS project officer 
explained the importance of the study and described the role it would play in the policy process. 
He also described in general terms what district participation would involve. The MPR 
representative explained the structure of the study, answered more detailed questions about what 
participation would involve, and addressed the primary concern about participation expressed in 
every district, confidentiality of data. Sometimes these discussions were conducted by 
telephone, sometimes in person. 

Initial contacts with candidate districts occurred in one of two ways: either a representative 
of the state education agency initiated the contact or the superintendent or business administrator 
in a neighboring pilot district initiated the contact. The decision as to which of these routes to 
follow was usually decided by the state education agency representative. 

The substance of the initial contact and associated follow-up also followed one of two 
patterns. In some instances, the official making the initial contact explained the study and 
secured agreement. The role of the study team was then to work out the details. In other 
instances, the person making the initial contact had the more limited role of introducing the idea 
of participation to senior district representatives, and securing their agreement to receive a visit 
or phone call from the study team to learn more about it. The study team then provided detailed 
information about the study, which senior district staff then used to make their decision about 
whether the district would participate in the study. 
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B. COMPARISON DISTRICT SELECTION AND RECRUITMENT FOR EACH 
PILOT DISTRICT 

In the course of selecting and recruiting comparison districts for the thirteen pilot districts 
remaining after St. Mary’s was excluded, we faced unique challenges with each case. These 
challenges could not always be adequately addressed using the general strategy described in 
Section A. In this section, we describe the details of the comparison district selection and 
recruitment process for each of the thirteen pilot districts. 



I. Blue Ridge School District (Pennsylvania) 

• Initially, there were 466 potential comparison districts in the restricted choice set. 

• Our QSI analysis generated a list of ten potential comparison districts for further 
consideration. 

• Discussions with state and local staff led us to select Montrose as the best choice for a 
comparison district. They were ranked U' on our original ranking of potential 
comparison districts. 

• To recruit Montrose, we relied on the superintendent from Blue Ridge to make our 
initial contact, which was successful. We then contacted Montrose to discuss the 
details of their involvement in the evaluation and they confirmed their commitment to 
participate. 



2. Stroudsburg, East Stroudsburg, and Pleasant Valley School Districts (Pennsylvania) 

• We grouped these three pilot districts together because of their proximity and 
similarity to one another in a number of dimensions. The similarity of the three was 
highlighted by the fact that our initial QSI ranking implied that for each, the other two 
pilot districts would have been selected as the top two comparison districts if they had 
not been a pilot district. 

• Initially, there were 464 potential comparison districts in the restricted choice set. 

• There was substantial overlap in the top ranked comparison districts for these three 
pilots. We identified 15 districts that jointly made up the toplO lists of the three pilot 
districts. 

• We gave this list of 15 potential comparison districts to knowledgeable state and local 
staff and also visited the pilot districts to discuss which of the comparison districts 
would be best. These conversations led us to identify the top three candidates 
(Bangor, Delaware Valley, and Pocono Mountain School Districts), along with 
several back-ups. 
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• We contacted the top three districts after an initial contact by a staff person at one of 
the pilot districts. Delaware Valley and Pocono Mountain School Districts declined. 
The status of Bangor was uncertain. We then contacted Easton, one of the back-ups, 
which agreed to participate. 

• One of the factors that local staff convinced us was important to consider in the 
matching process and that was not on our original list of QSI components was 
population growth. The area of eastern Pennsylvania in which the pilot districts are 
located grew more rapidly than the state as a whole during the 1990s. Because not all 
of our top ranked potential comparison districts were in equally high growth areas, we 
consulted with experts on Pennsylvania development and re-considered potential 
comparison districts located in two other high-growth areas in the state — 
southeast/south central Pennsylvania and the area near Pittsburgh. Although these 
districts had not been highly ranked in the original QSI ranking because of the 
proximity component, we re-examined the data to assess their comparability with 
respect to QSI components other than proximity. 

• This process led to a number of new candidates, and we made initial contacts with 
several districts in southeast Pennsylvania near the Maryland border. However, none 
of these districts proved to be suitable as a match for any of the pilot districts, and in 
the end we turned back to our original list of highly ranked comparison districts. 

• We then contacted Bangor again to confirm its willingness to participate. We also 
contacted Pottsgrove School District, which was on the original high ranking 
candidates based on the QSI, and Pottsgrove agreed to participate. 

• The three comparison districts ultimately selected and recruited are well matched to 
the three pilot districts in terms of the district characteristics used as QSI components. 
However, these districts do not as closely match the pilots in terms of their growth 
rates throughout the 1990s, as well as between 1999 and 2001. 



3. Maplewood School District (Ohio) 

• Initially, there were 567 potential comparison districts in the restricted choice set. 

• Our QSI analysis generated a list of ten potential comparison districts for further 
consideration. These ten districts were very well matched to the pilot district, with 
the tenth potential comparison district on the list not that much more different from 
the pilot district than the first potential comparison district in terms of the district 
characteristics used as QSI components. 

• Discussions with state and local staff led us to select a top ranked candidate district 
along with several back-up choices. 

• State officials contacted the top candidates (South Range School District and 
Matthews School District), but these districts declined to participate in the evaluation. 

• We then turned to our third choice for the evaluation — Newton Falls School District. 
With assistance from the FNS Midwest regional office and state officials, we 
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arranged a visit to Newton Falls School District and requested their participation in 
the evaluation. Newton Falls agreed to participate. They were ranked 8* on our 
original QSI ranking of top candidates. 



4. Salem City School District (Ohio) 

• Initially, there were 567 potential comparison districts in the restricted choice set. 

• Our QSI analysis generated a list of ten potential comparison districts for further 
consideration. Again, these ten districts were very well matched to the pilot district, 
with the tenth potential comparison district on the list not that much more different 
from the pilot district than the first potential comparison district in terms of the 
district characteristics used as QSI components. 

• Discussions with state and local staff led us to select a top ranked candidate district 
along with several back-up choices. 

• State officials contacted the top candidates, Beaver School District, but this district 
declined to participate in the evaluation. The FNS project officer and MPR project 
director contacted Niles School District, and that district also declined to participate. 

• We then turned to our third choice for the evaluation — Lisbon School District. With 
assistances from the FNS regional office and state officials, we arranged a visit to 
Newton Fall School District and requested their participation in the evaluation. 
Newton Falls agreed to participate. They were ranked on our original QSI 
ranking of top candidates. 



5. Creve Couer School District 76 (Illinois) 

• Initially, there were 211 potential comparison districts in the restricted choice set. 
This choice set was restricted to K-8 districts. 

• Our QSI analysis generated a list of ten potential comparison districts for further 
consideration. 

• We held a discussion of potential comparison districts with the food service director 
at Creve Couer. We did not send a list of the top ten potential comparison districts to 
her in advance, but presented the names of the best few matches to her orally. This 
discussion led us to identify one district that was clearly the best match — North Pekin 
and Marquette Heights School District 102 — along with two back-ups. These choices 
were also confirmed through discussions with the Illinois State Child Nutrition 
Director. North Pekin/Marquette Heights was ranked 7* on our original QSI ranking. 

• With the assistance of the food service director at Creve Couer, we contacted North 
Pekin & Marquette Heights and they agreed to participate in the evaluation. 
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6. Oak Park and River Forest School District 200 (Illinois) 



• Initially, there were 7 1 potential comparison districts in the restricted choice set. The 
relatively small number of districts in the choice set was due to the fact that Oak Park 
and River Forest is a single high school district. 

• None of the top ranked districts from our initial QSI analysis seemed appropriate as a 
match for Oak Park and River Forest. The main matching problem was that Oak 
Park/River Forest had an unusual combination of three characteristics: (1) a low 
baseline poverty rate (3 percent), (2) low certification rate (less than 10 percent 
certified for free meals) and (3) a relatively large proportion of black students (31 
percent). The districts that matched well on the poverty rate and certification rates 
tended to have very small proportions of students who were black. 

• We next held discussions with both state officials and the pilot district food service 
director. These discussions led us to identify a potential comparison district — 
Evanston Township High School District 202 — that appeared to be a relatively good 
match. Evanston had a much larger proportion of black students than the top ranked 
comparison districts. It was ranked 37^*’ in the original QSI ranking. Although its 
poverty rate and certification ratio were substantially higher than those of Oak Park 
and River Eorest, its ratio of the free certification rate to the poverty rate was close to 
that of Oak Park/River Eorest. In addition, the pilot district’s food service direct or 
felt that Evanston was similar to Oak Park/River Eorest in several other respects. 
Thus, we contacted Evanston and asked them to participate in the evaluation. 
Although they initially expressed interest in participation, they ultimately declined. 

• Next, we visited the Chicago area and visited several potential comparison districts. 
These visits along with discussions with state and local officials led us to identify four 
additional comparison districts that would make adequate matches for Oak Park/River 
Eorest. We approached these four districts — Zion Benton Township High School 
District 126, Niles Township High School District 219, Homewood Elossmoore High 
School District 233, and Mundeleine Consolidated High School District 120 — all 
declined to participate as well. 

• We then visited a meeting of suburban Chicago area food service directors, hoping to 
recruit two districts that were not on our original list of candidates. These two 
districts — Arlington Heights Township High School District 214 and Palatine 
Township High School District 21 1 — also declined our invitation to participate. 

• Einally, we expanded our search of possible comparison districts to include districts 
that were not just high-school-only districts. In particular, we began looking for K-12 
districts that included high schools whose characteristics were similar to those of Oak 
Park/River Eorest High School. Through this approach, we identified Boling Brook 
High School in the Valley View School District 365U. With assistance from the 
State Child Nutrition Director and the ENS Midwest Regional Office Special 
Nutrition Program Director, we secured their participation. 
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7. Williamson County School District (Tennessee) 



• Initially, there were 123 potential comparison districts in the restricted choice set. 

• Our QSI analysis produced only one potential comparison district — Wilson County 
School District — that appeared to be a good match for Williamson County. Other 
potential comparison districts either were smaller or had much higher poverty and 
certification rates. 

• Independent of our QSI analysis, we held discussions with state and local officials 
during a visit to Williamson County to get their views on which district in the state 
was most similar to Williamson. Without seeing our ranking, they also identified 
Wilson County as the most appropriate comparison district. 

• A representative of the Tennessee state education agency approached Wilson County 
and they agree to participate in the evaluation. Although the pilot project was 
implemented in 9 of 21 schools in the Williamson County School District, it was 
impractical to identify and recruit a matching subset of schools in Wilson County and 
the entire Wilson County district served as the comparison for the pilot schools in 
Williamson County. 



8. Dil worth- Glyndon-Felton School District (Minnesota) 

• Initially, there were 275 potential comparison districts for Dilworth-Glydon-Felton 
(DGF) in the restricted choice set. 

• Our QSI analysis generated a list of ten potential comparison districts for further 
consideration. 

• We spoke with staff in DGF about our the few potential comparison districts at the 
top of our rankings. We identified Lake Park Audubon School District, a small 
district east of DGF, as the best match. Discussions with staff there revealed that 
while Lake Park Audubon was similar to Glyndon and Felton, the easternmost towns 
in the DGF district, it did not have the same incidence of low-income and Hispanic 
families as Dilworth, due to its proximity to the Red River Valley where sugar beet 
farming is a major industry and employer of migrant laborers. This led us to invite 
East Grand Forks School District, which had a comparable population of these 
groups, to participate, but they declined. 

• We then decided to try to use two districts jointly as a comparison for DGF, one 
located on the Red River (and thus similar to Dilworth) and one located to the east of 
the river (and thus more similar to Glyndon and Felton). Using two districts also had 
the advantage of reducing for any one district the percentage of students whose 
families would need to be asked to participate in the study, a source of particular 
concern to the officials from these small school districts. This strategy led us to keep 
Lake Park Audubon (east of the river) as a comparison district and to select 
Breckenridge (on the Red River). 
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We approached these districts with this newly developed proposal to participate in the 
evaluation and they both agreed. 






9. Dunkirk City School District (New York) 

• Initially, there were 600 potential comparison districts in the restricted choice set. 

• Our initial QSI analysis generated no potential comparison districts that appeared to 
be suitable as matches for Dunkirk. The problem was the combination of being a 
relatively small district but having a large Hispanic population (32 percent of 
students) and high poverty and certification rates. The top ranked districts all had 
much smaller Hispanic populations, while districts with comparable Hispanic 
populations were far away. In particular, these districts were located in the Hudson 
River Valley or near New York City. Discussions with local experts led us to 
conclude that these far-away districts would have little face validity as comparisons to 
Dunkirk. 

• Another strategy we employed to try to locate a comparison district was to expand the 
choice set to include districts located in parts of Pennsylvania or Ohio that were 
reasonably close to Dunkirk. We found several districts in northeast Ohio that 
matched Dunkirk in the proportion Hispanic and several other dimensions but each 
also differed from Dunkirk along some important dimension. 

• Ultimately, we decided that Jamestown City School District, a district close to 
Dunkirk, would be the most appropriate comparison district. Although Jamestown 
had a much smaller Hispanic population than Dunkirk (7 percent), its Hispanic 
population was larger than those of other top ranked comparison districts and it was 
also well matched in terms of its poverty, certification, and participation rates (it was 
ranked 7* in our original QSI ranking). It also had substantial face validity in the 
eyes of local officials. Part of our decision to invite Jamestown to participate 
involved a strategy in which we would oversample Hispanic students in Jamestown 
so that their representation in our students samples would more closely match that of 
Dunkirk. 

• A representative of the New York State Education Agency contacted Jamestown City 
on behalf of the study and they agreed to participate. 



10. Grandview Consolidated School District Number 4 (Missouri) 

• Initially, there were 289 potential comparison districts in the restricted choice set. 

• Our QSI analysis generated only three districts that we felt were suitable as 
comparison districts to Grandview. Grandview School District is a district within the 
Kansas City metropolitan area, and approximately half of the district students are 
African American. The three top ranked potential comparison districts were similarly 
situated, but the districts after the top three had predominantly white student 
populations. 
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• In an effort to identify a larger pool of potential comparison districts for further 
consideration, we began considering districts outside of close proximity to 
Grandview. In particular, we examined districts in the St. Louis metropolitan area. 
This process led us to identify three additional districts whose original QSI ranking 
was low because of the proximity component but that were otherwise similar to 
Grandview. 

• We presented the list of these six potential comparison districts to state officials and 
discussed the suitability of these districts. These discussions led us to identify one of 
the Kansas City metropolitan area districts — Hickman Mills C-1 School District — as 
the best match. 

• With the assistance of the Missouri state officials, we approached Hickman Mills and 
they agreed to participate in the evaluation. 



11. Morenci School District (Arizona) 

• Initially, there were 87 potential comparison districts in the restricted choice set. 

• Our initial QSI analysis generated no districts that would be suitable comparison 
districts. The features of Morenci that made it a difficult pilot district to match were 
the combination of a very high Hispanic population (54 percent), a fairly low poverty 
rate (10 percent), and a very low certification rate (10 percent certified for free 
meals). In addition, Morenci is a small mining town in a rural area of Arizona. Other 
potential comparison districts that matched Morenci in terms of its poverty and 
certification rates had smaller Hispanic populations. Those with similar Hispanic 
populations tended to have much higher poverty rates. 

• In an attempt to find a suitable comparison district for Morenci, we spoke with the 
Morenci superintendent and food service director and with state officials. After 
considerable discussion they agreed with our judgment that none of our top ranked 
comparison districts would make credible comparison districts. They suggested that 
we examine the school districts in other small mining towns in Arizona. We 
identified five or six such school districts, but none of these districts was a good 
match for Morenci, since most of them had high poverty and certification rates. 

• Next, we expanded our search to school districts in neighboring states. First, we 
examined districts in mining towns in New Mexico, but these districts had the same 
limitations as those in Arizona. We then conducted a wider search across all school 
districts in New Mexico and Colorado. This search did not identify any districts that 
matched the characteristics of Morenci very well. 

• Because the QSI analysis failed to produce any good matches for Morenci, and 
because alternatives suggested by our subsequent discussions and analysis were 
uniformly believed to be not credible by individuals familiar with the area, we 
suspended our search for a comparison district in Arizona, and dropped Morenci from 
the evaluation. 
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III. SAMPLE DESIGN AND SAMPLE SELECTION 



This chapter describes the sample design and sample selection process. 

A. SAMPLE DESIGN 

The pilot districts each volunteered to test one of several approaches to modifying the 
application and verification process that FNS designed. Because of the voluntary nature of 
district participation, the pilot districts are self-selected and not a representative sample of all 
districts nationwide. As described in the previous chapter, the comparison districts were 
carefully matched to each pilot district, and therefore are also not representative of districts 
nationwide. 

Each pilot district and its matched comparison district are essentially a “case study.” The 
test of Up-Front Documentation is essentially nine case studies; the test of Graduated Verification 
is three. We measure the average impact of Up-Front Documentation as the average of the 
impacts across the nine site pairs, and the average impact of Graduated Verification as the 
average of the impacts across the three site pairs. The sample of students was designed with this 
analytic goal paramount. 

I. Summary of the Design 

In each of the 25 study districts, the household survey sample was selected to represent all 
students enrolled at the beginning of school year 2002-2003 who were (1) not approved for free 
meals through direct certification in school year 2002-2003, and (2) enrolled in the district at the 
time of the data collection in October to December 2002. The sample for the household 
interviews was stratified into 75 strata by district (25) and meal price status (3). 
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We selected each district sample from lists of students who were approved for free meals by 
application and approved for reduced-price meals and from lists of all students enrolled at the 
start of school year 2002-2003. We then combined and unduplicated these lists to form three 
sampling strata for each district: 

1 . Students who are approved for free meals but not directly certified 

2. Students approved for reduced-price meals 

3. All other students, or the balance of the district population 

Because we conducted sample selection using lists provided near the beginning of the school 
year 2002, the meal price status used for sample selection did not in all cases reflect the meal 
price status according to the SFA on October 31, 2002. Therefore, after survey data collection 
was conducted, we determined each student’s final meal price status by matching our sample 
members against a meal price status list that reflected each student’s status on (or about) October 
31, 2002.^ Table III.l shows target sample sizes by meal price status and district stratum, 
released sample by sampling frame meal price stratum, and released samples by final meal price 
status. 

The study objective is to draw inferences about all students in the study districts, excluding 
students directly certified, and our sampling objective is to select a representative sample. 
However, because students are embedded in households, we needed to decide how to handle the 
situation in which a household contains more than one student. One consideration is that the 
income eligibility of students in the same family is the same. Furthermore, keeping the interview 
simple and the burden on households low required asking parents to report about their income 
one time and to report on the lunch programs experiences of one child. We decided to select 

'chapter VI describes this post-sample frame matching and provides related population 
estimates. 
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SUMMARY OF STUDENT SAMPLE SIZES 
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and interview one student per household. Therefore, we grouped students in households and 
selected one per household for the student sample. Sample weights used in the analysis account 
for the higher selection probabilities of students from households with more students. 

We considered varying the allocation of sample to districts and meal price groups within 
districts in a manner that satisfied variance constraints on key outcomes and took account of the 
differing costs of data collection among families whose incomes fell above and below 
400 percent of the federal poverty level (FPL). Using sample optimization methods (described 
in the next section), we determined that equal samples per district within each pilot type were 
close to optimal.^ Because equal allocations by district and meal price status group greatly 
simplified both the sampling process and survey operations, we decided to use equal allocations 
rather than the optimal ones. 

2. Analysis of Sample Allocation 

Several factors in addition to schedule created a very challenging problem for stratification 
and allocation of the sample. This study is unusual because the research questions focus on 
subgroups of households defined in terms of their income for household size and meal price 
status rather than on the population as a whole. For example, the deterrence issues focus on 
students whose family incomes are above 185 (or 130) percent of poverty; the barrier issues 
focus on students whose family incomes are below those thresholds; and issues surrounding 
accuracy focus on students who are certified for free or reduced-price meals (and asks what 
percentage of them are above or below the eligibility thresholds). While these subgroups were to 
be identified using the data on household income obtained through the survey, the information 

In the end, we used different sample sizes in some districts after the final comparison 
districts were selected. 
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available before the survey for sample stratification was only indirectly related to the analytic 
groups. Furthermore, the analytic groups overlap in such a manner that the number of sample 
points allocated to one group affects the number allocated to others. Consequently, the sampling 
process had to assign differential sampling rates to various groups in a way that ensured a 
sufficient sample size in each domain while avoiding allocating more than necessary to other 
domains. Finally, because the interview costs differed by household income level, efficient 
sample allocation had to take these cost differences into account. 

To assess alternative allocations, we assumed that we would obtain from each participating 
district a list showing each student’s meal price and then select a single-stage stratified sample 
under a specified allocation plan. We first defined sampling strata that would correspond as 
closely as possible to the key analytic groups. Because the available lists contained information 
on district membership and meal price status, we planned originally to stratify the sample into 84 
explicit sampling strata based on district membership (28 total, made up of 10 Up-Front 
Documentation districts plus their matched comparison districts, and four Graduated Verification 
pilot districts plus their matched comparisons) and the student’s meal status (approved for free 
meals [excluding direct certified], approved for reduced-price, and remaining non-meal program 
students designated as “paid”). The final sample included 25 school districts, with three meal 
price categories, for a total of 75 sampling strata.^ 

Because the strata did not correspond to the analytic subgroups, the second major step was 
to estimate the percentage of each stratum group that belonged to each analytic group. This 

The RFP called for including 14 pilot SAFAs in the evaluation of these two were dropped 
(Morenci, AZ and St. Mary’s in Paterson, NJ). Eleven of the included pilot districts were 
matched to a single comparison district. For the Dilworth Glyndon-Felton School District in 
Minnesota two comparison districts were selected for a total of 25 districts. 
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mapping from sampling strata to analytic groups allowed us to estimate how changes in the 
allocation of the sample across the strata would affect sample sizes for each analytic group. We 
used data provided by FNS about the rate of benefit reduction and termination in the standard 
verification process during the pre-pilot period in the pilot districts to estimate these mappings."^ 
The final strata profiles used for assessing sample allocation are shown in Table III.2A and Table 
III.2B, which also show initial estimates of the population counts in each stratum for each 
analytic group and the percentage of these cases represented in each stratum. Because the 
available district level data were insufficient to support district-specific estimates, we assumed 
that the mapping would be the same across all districts in each pilot type. 

An example from the Up-Front Documentation pilot districts will help to clarify the 
structure of Table III.2A and Table III.2B. We first estimated the percentage of each stratum 
group in each analytic group. For example, for the analytic group “income >185 percent FPL,” 
we estimated that 19 percent of free-certified students, 19 percent of reduced-price-certified 
students, and 91 percent of paid students would have income >185 percent FPL. Second, 
applying these percentages to the population counts for free, reduced-price, and paid produced 
estimated populations of 535 free-certified with income >185 percent FPL, 327 reduced-price 
certified with such income, and 23,053 paid with such income. Finally, from these estimated 
populations we computed the percentage of each analytic group belonging to each stratum group: 
among students with income >185 percent FPL, 2.2 percent were estimated to be free, 1.4 
percent reduced-price, and 96.4 percent paid.^ 

"^Modification to RFP FNS-02-008JMC transmitted from Michael Rich to Jim Ohls on 
February 6, 2002. 

^The distribution by stratum within analytic group helped to show how allocating more (or 
less) sample to the free stratum affected the number of sample in each analytic group. 
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TABLE III.2A 



EXPECTED MAPPING OF SAMPLING STRATA INTO THE ANALYTIC GROUPS— 
UP-FRONT DOCUMENTATION 







Stratum Group 
















Percentage 
of Population 










Total 


in Analytic 


Analytic Group 


Free 


Reduced 


Paid 


Population^ 


Group 


Population Count 


2,816 


1,719 


25,333 


29,868 




Percentage of Population 


9.4 


5.8 


84.8 


100 




Certified Free Not Directly Certified 












Percentage of Stratum Group in This Analytic Group 


95 


4 


1 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


2,675 


69 


253 


2,997 


10.0 


Group 


89.3 


2.3 


8.5 


100.0 




Certified Reduced-Price 












Percentage of Stratum Group in This Analytic Group 


2 


90 


1 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


56 


1,547 


253 


1,857 


6.2 


Group 


3.0 


83.3 


13.6 


100.0 




Income >185 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


19 


19 


91 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


535 


327 


23,053 


23,914 


80.1 


Group 


2.2 


1.4 


96.4 


100.0 




Income >130 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


39 


92 


97 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


1,098 


1,582 


24,573 


27,253 


91.2 


Group 


4.0 


5.8 


90.2 


100.0 




Income <130 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


61 


8 


3 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


1,718 


138 


760 


2,615 


8.8 


Group 


65.7 


5.3 


29.1 


100.0 




Income <185 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


81 


81 


9 






Estimated Population 

Percentage of Analytic Group in Each Stratum 


2,281 


1,393 


2,280 


5,953 


19.9 


Group 


38.3 


23.4 


38.3 


100.0 





■‘Population Counts include all students not directly certified in the 10 Up-Front Documentation districts specified for 
inclusion in the REP for the study. Data pertain to the year prior to each districts’ first year of pilot operation. Figures 
are totals across districts in the group. 
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TABLE III.2B 



EXPECTED MAPPING OF SAMPLING STRATA INTO THE ANAEYTIC GROUPS— 

GRADUATED VERIFICATION 



Analytic Group 


Free 


Stratum Group 
Reduced Paid 


Total 

Population 


Percentage of 
Population in 
Analytic 
Group 


Population CounC 


2,052 


751 


4,306 


7,109 




Percentage of Population 


28.9 


10.6 


60.6 


100 




Certified Free Not Directly Certified 












Percentage of Stratum Group in This Analytic Group 


95 


4 


1 






Estimated Population 


1,949 


30 


43 


2,023 


28.5 


Percentage of Analytic Group in Each Stratum Group 


96.4 


1.5 


2.1 


100.0 




Certified Reduced-Price 












Percentage of Stratum Group in This Analytic Group 


2 


90 


1 






Estimated Population 


41 


676 


43 


760 


10.7 


Percentage of Analytic Group in Each Stratum Group 


5.4 


88.9 


5.7 


100.0 




Income >185 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


19 


19 


91 






Estimated Population 


390 


143 


3,918 


4,451 


62.6 


Percentage of Analytic Group in Each Stratum Group 


8.8 


3.2 


88.0 


100.0 




Income >130 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


39 


92 


97 






Estimated Population 


800 


691 


4,177 


5,668 


79.7 


Percentage of Analytic Group in Each Stratum Group 


14.1 


12.2 


73.7 


100.0 




Income <130 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


61 


8 


3 






Estimated Population 


1,252 


60 


129 


1,441 


20.3 


Percentage of Analytic Group in Each Stratum Group 


86.9 


4.2 


9.0 


100.0 




Income <185 Percent FPL 












Percentage of Stratum Group in This Analytic Group 


81 


81 


9 






Estimated Population 


1,662 


608 


388 


2,658 


37.4 


Percentage of Analytic Group in Each Stratum Group 


62.5 


22.9 


14.6 


100.0 





■‘Population counts include all students not directly certified in the 3 Graduated Verification districts specified for 
including in the REP for the study. Data pertain to the year prior to each district’s first year of pilot operation. Figures 
are totals across districts in the group. 
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To be optimal, or nearly so, for this study’s objeetives, the sample alloeation had to meet 
preeision objeetives for three analytie groups: (1) students approved for free meals by 
applieation, (2) students in households with ineomes above 185 pereent of the FPL, and 
(3) students in households with ineomes of 185 pereent or less of the FPL.^ Within these three 
analytie groups, it was important to be able to deteet differenee in proportions between the 
pooled pilot and eomparison distriets of 10 to 12 pereentage points or less. This required 
oversampling some groups and undersampling others, whieh ereated design effeets due to 
differential seleetion probabilities and redueed the effeetive sample size for any nominal sample 
size generated.’ Aeeordingly, for eaeh alloeation plan tested, we ealeulated the eorresponding 
preeision levels, given the design effeets ereated by the speeifie sample alloeation plan. 

In addition to the general preeision objeetives for designing an optimal alloeation, we 
needed to eonsider the data eolleetion eosts and the planned pooled distriet data analysis. In this 
study, the eost of eondueting the survey varies with the alloeation plan, beeause the paid strata 
were expeeted to have a lower eost per eomplete, as they would eontain more households with 



^Table V.2A and Table V.2B present our analysis of sample preeision for the final design 
and eompare planning targets with aetual preeision aehieved. 

’The design effeet due to unequal seleetion probabilities in a stratified sampling plan is the 
ratio of the variability in the estimates that is obtained from a speeifie sample alloeation relative 
to the variability that would result with a sample that is alloeated in proportion to the strata 
population sizes. For example, a design effeet of 1.2 indieates that the design seleeted introduees 
20 pereent more variation in the survey results than would a design that uses simple random 
sampling with a proportional alloeation. Dividing the sample size by the design effeet 
determines the effeetive sample size that should be used to gauge the preeision level of the 
estimates using simple random- sampling-theory-based eonfidenee intervals. We note that in the 
absenee of aetual survey outeomes, one ean estimate a survey design effeet due to weighting by 
eomputing the l-i- CV^ of the weights (the square of the eoeffieient of variation, whieh is equal 
to the standard deviation of the weights divided by the mean weight value). This approximation 
to the true design effeet estimates the relative inerease in the variation in the survey outeomes 
from the planned sample alloeation, but it does not aeeount for the effeets of stratifieation (whieh 
would reduee the design effeet) or elustering (whieh would inerease it). 



29 




incomes over 400 percent of the FPL that would not require a part 2 interview. Therefore, while 
a variety of allocation schemes may yield the desired precision levels, we wanted to find the one 
that minimized the survey costs. 

To account for these constraints, we used a mathematical technique based on a methodology 
developed by Chromy (1987). The allocation algorithm can be described as a two-step process 
for a stratified non-geographically clustered sample directed to: 

(1) Minimize cost (Cj = 27 c /, n h 

(2) Subject to precision constraints (Var*k ) on variance estimates (14 ) 

VaTk < Var*k where Vark = S {W^ Varkh } / {nh Pk,h} 

where Var*t is the statistical precision constraint for the kth estimate (in the algorithm this 
constraint is entered as the value of the desired standard error of the estimate being the square 
root of the value of Var*t ), the Vart_hS are the estimated variance components (the expected 
variability in the outcomes) associated with the kth estimate and the hth stratum, and these are 
then weighed by relative stratum population counts, and divided by a sample size factor to 
sum to the resulting sampling variances for each domain Vart . The sample size factor, rih Pk,h, 
takes into account that only a portion of the sample selected, nn, in our situation from stratum 
h will be a member of analytic group k. 

To apply the method for our application, we composed the variance components to reflect 
variation in a binomial outcome. We defined this statistic to have a mean value of p, implying a 
variance equal to p*(l-p). Across the six analytic groups the value of p with the smallest 
variance was .1, and the value of p with the largest variance was .63. Values assumed for the 
comparison districts from which the pilot impacts would be measured are presented in Table 
V.2A and Table V.2B. Given the plan to compare equally weighted averages of the district 
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outcomes, we set up the algorithm so that h above indexed the originally specified 42 districts by 
meal status combinations for the pilot districts and k defined the combinations of 14 districts by 

o 

the six analytic groups (84 constraints). Separately for the Up-Front Documentation and 
Graduated Verification study districts, we specified that the precision for each district should be 
equal and have an average value that would yield the desired minimum detectable difference 
(MDD) between the pilot and comparison districts. For example, all 10 originally specified Up- 
Front Documentation pilot districts received the same precision constraint for each analytic 
group. Therefore, when these districts were pooled using an equally weighted average approach, 
that would produce a standard error of about .035 (translating to an individual district standard 
error constraint of .246), which would equate to an MDD of 10 percentage points at 80 percent 
power. While the algorithm did not allow us to specify a constraint that the district sample sizes 
within each program should be equal, we believed this equal-by-district variance constraint 
model would nearly achieve that objective and would form the constraints on the variances to 
parallel the planned analysis. We set the values of to be equal across districts, but to vary by 
meal price status. For the values we used the district-by-meal-status population counts as 
presented in the RFP. Finally, for the paid stratum group in each district, we assumed the 
average interview cost would be about one-half that of the cost of interviewing in the other strata 
since we would not have to conduct in-person interviews with those having incomes more than 
400 percent of the FPL. 

We conducted several exploratory applications of the sample optimization model. We 
initially imposed an MDD of 10 percentage points or lower between the combined comparison 

o 

As noted above, the RFP for the study specified 10 Up-Front Documentation districts and 
four Graduated Verifications. These specifications were used in sample planning. 
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site estimate and the combined pilot site estimate at 80 percent statistical power and 95 percent 
confidence for each of the three key analytic groups. For each of the remaining objectives, we 
set less-stringent requirements — MDDs of 12 to 15 percentage points. We then began relaxing 
constraints as needed until we obtained an overall sample size of about 2,800. The final 
allocation suggested a total of 1,740 interviews to be allocated across each of the 10 Up-Front 
Documentation pilot district and their comparison districts (87 per district) and 1,040 across the 
four Graduated Verification pilot districts and their comparison districts (130 per district). 

Because the methods were applied at the district level and accounted for differences among 
districts in the population distribution by meal price status, the algorithm did not produce an 
optimal sample allocation in which sample sizes were equal across districts. However, the 
differences by district were small for the optimum allocation. We used the algorithm^ to 
compute how the precision would change if each district’s optimal total sample size was adjusted 
so that sample sizes were equal by district within each set of pilots. Adjusting the totals to be 
equal by district had very small effects on the precision. As a final step, we evaluated the effects 
of allocating equal numbers of free, reduced-price, and paid students in each district, and found it 
to have only a small impact on the precision (it increased the MDDs by about 1 percentage 
point). Accordingly, we set the sample sizes at 19 free, 14 reduced-price, and 54 paid (87 total 
per site) in each Up-Front Documentation pilot and comparison site, and at 57 free, 21 reduced- 
price, and 52 paid (130 total) for the Graduated Verification pilot and comparison sites. 

As planning proceeded, two pilot sites were excluded from the evaluation: Morenci School 
District in Arizona (an Up-Front Documentation plot district) and St. Mary’s School in Paterson, 

^ The algorithm we used was designed to allow the user to specify a starting or alternative 
sample allocation for which it computes the resulting precision estimates for comparison to the 
optimal allocation suggested to meet the specify precision targets. 



32 




New Jersey (a Graduated Verification pilot district). To compensate for this reduction in the 
number of districts from 28 to 24, we increased the sample sizes in each remaining district. The 
final allocation consisted of 21 free by application, 16 reduced-price, and 60 paid (97 total) per 
Up-Front Documentation district. In the Graduated Verification districts, the sample sizes varied 
from 31 free, 11 reduced-price, and 28 paid (70 total) to 76 free, 28 reduced-price, and 69 paid 
(173 total). 

B. DESCRIPTION OF THE SAMPLE SELECTION PROCESS 

Forming the sampling strata required placing all students enrolled in each participating 
district into four categories: (1) directly certified for free meals, (2) certified for free meals by 
application, (3) certified for reduced-price meals, and (4) not certified for free or reduced-price 
school meals. Nineteen districts provided four separate lists, which included (1) students 
approved for free meals, (2) students approved for reduced-price meals; (3) students approved 
for free meals by direct certification (if applicable), and (4) a full enrollment listing for the 
current school year. We constructed the sample frame by identifying and eliminating duplicate 
entries and then merging the meal price status list with the full enrollment list. Six districts 
provided a full enrollment listing with meal status information, which included meal price status 
information for each student, and thus provided the necessary sampling frame. 

Sample selection entailed a two-, three-, or four-stage stratified process, depending on the 
nature of the sample frame and whether the district chose to use a passive consent process and/or 
provide contacting information only for sampled students. Stage 1 used an interval sampling 
method to reduce the size of hard-copy lists for data entry. Stage 2 used stratified systematic 
sampling to reduce the list/sample resulting from Stage 1 for five districts that requested limiting 
the contact information provided to a sample of their student body. After grouping students 
selected in stages 1 and 2 who lived in the same household. Stage 3 consisted of selecting a 
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sample of households. Stage 4 involved selecting one student from each household group 
selected in Stage 3. 

We based the stratification for each of the sampling stages 2 through 4 on district 
membership and meal price status. In one Graduated Verification comparison district, the 
sample was also stratified by Hispanic surname, and students with Hispanic surnames were 
oversampled in order to align the sample’s ethnic composition better with that of its pilot district. 

To achieve the desired study schedule, we conducted the sampling frame development, the 
sample selection, and the interviewing process on a flow basis as we received information from 
the districts. In the next two subsections, we provide details on the list acquisition and 
processing and sample selection steps. 

1. List Acquisition and Processing 

As noted, most districts furnished three or four different lists of students from which we 
developed sampling frames. Table III.3 provides an overview of the sampling lists received 
from each district. It shows (I) the medium of the full enrollment list (whether in electronic 
format or in hard copy) and, if in hard copy, the sampling interval used to select an initial sample 
for data entry; (2) the medium of the list of students approved for free and reduced-price meals; 
(3) the point at which the meal price status list was generated (prior school, current school year 
before 30th day, current school year after 30th day); (4) the medium of the list of directly 
certified students (if applicable); (5) the variables used to match the free and reduced list to the 
direct certification list; and (6) the variables used to match the meal price status list to the full 
sample. All the direct certification lists and the full enrollment lists reflected the status of 
students at the start of the 2002-2003 school year. 
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SAMPLING FRAME SUMMARY: CHARACTERISTICS OE LISTS PROVIDED AND PROCESSING METHODS 



Variables Used to Match 


Meal Price Status to Full 
Enrollment List 




Q 


Q 


DC: Name and birth date, 
F&R: Name, grade 


Name, grade | 


Name, grade | 


NA 1 


Name, grade, school | 


Name, street number | 


Name only | 


Name, grade | 


Name (grade partial) | 


Name, grade | 


NA 1 


Name, grade | 


Name only | 


NA 1 


NA 1 


Name (grade partial) | 


Free and 
Reduced List to 
Direct Cert. List 




NA 


Q 


NA 


Name (No dups) 


Name and grade 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


List of 
Directly 
Certified 
Students 


Medium; 

HC = 
Hardcopy, 
Elec. = 
Electronic 




NA 


HC 


Elec. 


HC 


HC 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


List of Students Approved 
for Free and Reduced-Price 
Meals 


Medium; HC 
= Hardcopy, 
Elec. = 
Electronic 




HC 


HC 


Elec. 


HC 


HC 


NA 


HC 


Elec. 


HC 


HC 


Elec. 


HC 


NA 


HC 


HC 


NA 


NA 


HC 


Status^ 




Oh 


Clh 




C>30 


Oh 


NA 


cu 


C>30 


Oh 


C<30 


C>30 


C<30 


NA 


Oh 


Oh 


NA 


NA 


C<30 


Full 

Enrollment 

List 


Medium 

(Sampling 

Interval); 

HC = 
Hardcopy, 
Elec. = 
Electronic 




HC (2) 


HC (4) 


HC (10) 


Elec. 


HC (6) 


Elec. 


HC (5) 


Elec. 


HC(1) 


HC(1) 


HC (2) 


HC (2) 


Elec. 


Elec. 


Elec. 


Elec. 


HC (2) 


HC(ll) 




Pilot 

Comparison 

Status 




CU 


U 




U 


Oh 


U 


cu 


U 




U 


Oh 


U 


cu 


U 


Oh 


U 


cu 


U 




Processing Order 


Up-Front Documentation 


Blue Ridge, PA 


1 Montrose, PA 


East Stroudsburg, PA 


1 Easton, PA 


1 Pleasant Valley, PA 


1 Bangor, PA^ 


1 Stroudsburg, PA 


1 Pottsgrove, PA 


1 Maplewood, OH 


1 Newton Falls, OH 


1 Salem, OH 


1 Lisbon, OH 


1 Creve Coeur, IL' 


1 North Pekin and Marquette Heights, IL 


1 Oak Park and Forest River, IL 


1 Valley View, IL^ 


1 Williamson County, TN“ 


1 Wilson County, TN^ 
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TABLE III. 3 {continued) 



Variables Used to Match 


Meal Price Status to Pull 
Enrollment List 




Name, grade 


Name only | 


Name, grade | 


Name, grade | 


DC-ID, P&R- ID 1 


NA 1 


Q 


Free and 
Reduced List to 
Direct Cert. List 




Name and grade 


NA 


NA 


Name and grade 


NA 


NA 


Q 


List of 
Directly 
Certified 
Students 


Medium; 

HC = 
Hardcopy, 
Elec. = 
Electronic 




HC 


HC 


NA 


HC 


HC 




HC 


List of Students Approved 
for Free and Reduced-Price 
Meals 


Medium; HC 
= Hardcopy, 
Elec. = 
Electronic 




HC 


HC 


Elec. 


HC 


HC 


NA 


NA 


Status^ 




Gh 


C<30 


C<30 


C<30 


C<30 


NA 


NA 


Full 

Enrollment 

List 


Medium 

(Sampling 

Interval); 

HC = 
Hardcopy, 
Elec. = 
Electronic 




HC(1) 


HC (2) 


HC (2) 


HC (5) 


HC(ll) 


HC (5) 


Elec. 




Pilot 

Comparison 

Status 




Gh 
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Gh 
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Gh 


U 




Processing Order 


Graduated Verification 
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“For Dunkirk, the full enrollment file identified the meal status: free and reduced; however the district provided only 1 in 5 sample but also provided total counts. 





Eight of the 25 districts provided electronic lists, which we converted to a standard SAS data 
set format with standardized data field names for subsequent processing. Seventeen of 25 
districts provided a full enrollment list in hard-copy format. The first processing step was to 
enter the data into electronic files. To reduce costs and perform the sampling more quickly, we 
selected a sample of each full enrollment list for data entry. Because we planned to match the 
meal price status lists to this full enrollment sample to produce a single sampling frame for the 
study, we set the sampling interval for the full enrollment list to ensure they would contain 
enough free and reduced-price students. We established the sampling interval by calculating the 
interval necessary to meet sample targets for the lowest incidence stratum, and then doubled the 
sampling rate to allow for loss due to nonresponse and other reasons. After setting the interval, 
we selected a random integer within the interval as the start point, and then selected every kth 
case (where k is the sampling interval). In most cases the hard-copy lists were ordered by 
student last name, which we felt would not introduce any periodicity effects (such as biases in 
the selection process) from the use of an interval sampling method. If districts provided the free 
and reduced-meal price lists in hard copy, we entered data for all students on these lists. 

The second major task in preparing the sampling frame was to match the free, reduced-price, 
and directly certified lists to the full enrollment lists. Nineteen of the school districts 
maintained the list of students approved for free and reduced-price meals separately from the full 
enrollment records and consequently could not provide a single list that met our needs. 

''^Of the 25 districts, 19 provided completely separate lists. The Jamestown (NY) and Creve 
Coeur (IL) school districts provided a list that identified free and reduced students but omitted 
direct certification, which they supplied on a separate list at a later date. The Bangor (PA), 
Valley View (IL), Williamson (TN), and Dunkirk school districts provided a list that identified 
meal status among the full enrollment (Dunkirk provided only a sample) for free, reduced, and 
direct certification. 
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Furthermore, only 3 of the 19 districts provided an identification number that allowed us to link 
the entries across the lists. Consequently, in 16 districts, we matched the lists showing the 
student’s meal price status to the full enrollment list (or the sample selected from it) for creating 
the sampling strata. In most districts, the information available to perform this match was 
limited to the student’s name and grade. 

We used an electronic matching process to classify the students by meal price status for 
sampling purposes. We chose this method for two reasons. First, we believed that extensive 
visual matching to create the sample frames would have created delays in the start of data 
collection, which would not have been acceptable given the tight schedule for completing the 
data collection effort. Second, we anticipated that electronic matching methods would be 
accurate enough for sample stratification purposes, which could tolerate some classification 
errors without seriously diminishing the benefits of stratification in the sample. Third, we 
planned to perform a visual matching of our sample to lists of students approved for free and 
reduced-price meals as of the end of October. For 16 of the districts, the meal price status lists 
that they provided in time for sampling were not final, and we acquired a more current list to 
support the data analysis. Because manual matching would have to have been repeated, we 
decided to limit our matching efforts in preparing the sampling frame and to expend the 
resources to conduct a more careful visual review on the updated lists received after October 31, 
2002 (discussed in Chapter IV). 

The first step in the matching process was to combine the free, reduced-price, and direct 
certification lists into a single file. While the free and reduced lists usually were provided 

"in 9 of the 16 districts, the lists provided for the free and reduced-price students were from 
the previous school year. In the other 7 districts, the lists were more than 30 days old at the time 
interviewing began. 
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separately, a sorting and review process on the records by name revealed little if any duplication, 
and therefore we were able to combine these without the need for a formal matching process. In 
some districts, as Table III.3 indicates, if the district used direct certification, direct certification 
status of the student was indicated on the free certified listing, and combining the free and 
reduced-price lists yielded a single list of students approved for free (non-directly certified), 
reduced and directly certified for free meals. However, four districts provided separate direct 
certification lists, of which one provided a linking identification number and the other three 
required a name and grade matching process (see below) to eliminate across-list duplication.^^ 
After preparing a combined free/reduced-price/direct-certification list, we matched this list to the 
full enrollment file to create the sampling frame. 

If name and grade were used to match student records electronically, we used a two-stage 
process in which the second stage helped mitigate the effects of misspellings and other errors in 
the data. For the first step, we identified via SAS those students on both files whose names 
matched exactly. We checked these to determine whether the grade level (when available — 3 of 
the 16 districts did not provide grade-level information on the meal status lists, and 2 others had 
high levels of missing grade data) was within one year and printed for visual review those cases 
in which the grade differed by more than one year. From the remaining names from each 
source, we then applied a SAS Soundex matching process'"^ to link any records that were true 

1 2 

Among the duplicates, membership on the direct certification lists took priority for the 
final classification. 

1 

In general, many of these sets had a missing grade value from one source (to which we 
considered them to be a match) or were clearly related siblings (which we considered to be a 
non-match). 

''^The SoundEx matching algorithms are methods of matching components that sound alike 
but have different spellings. Both methods are fairly effective in locating misspellings, 
particularly those due to typographical errors. 
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matches the exact matching process had missed because of typographical errors or misspellings. 
We then applied the same review of grade level as was used for the exact matches to finalize the 
meal status for sampling. We used these methods in 3 districts to match the free and reduced- 
meal list to the direct certification lists and in 16 districts to match the combined meal price 
status listing to the full enrollment list/sample. Ultimately, linked records on the full enrollment 
list received the meal status from the combined free/reduced/direct certification file. We then 
removed directly certified students from the enrollment list to finalize the sampling frame. 

Table III.4 shows sampling frame counts by district, and the estimated proportion of 
students certified for free and reduced-price meals that we could match to the current year 
enrollment list/sample. The match rate is the total number actually matched divided by the 
expected number that would match (the number on the resolved list of students certified free and 
reduced-price multiplied by the initial rate at which names were selected from the full enrollment 
list for initial data entry). 

In general, the match rates were lower than we hoped for, with a low of 61.2 percent in 
Grandview, MO and a match rate below 90 percent in 13 of the 25 districts. Among these 13 
districts, 8 provided an initial listing of meal price status for the prior school year. Use of a prior 
year caused new entrants to the district’s schools (including new kindergarten students) to be 
assigned a status of paid, which for some was almost certainly incorrect and reduced the 
efficiency of the sample stratification. For the other five districts, including Grandview, MO, we 
could not identify a reason for the low matching rate. 



'^For example, if we conducted a l-in-5 sampling process on the full enrollment list prior to 
data entry, we would have expected only one-fifth of the meal status students to have linked 
successfully to the full enrollment sample. 
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As a check on our electronic two-step matching procedures, we manually reviewed the 
listings in several districts and found that many of the students on the free and reduced-price list 
simply were not listed on the full enrollment list. This suggested to us that the free and reduced- 
price lists include some students who no longer attend district schools, and therefore the actual 
percentage of students certified for free and reduced-price meals is lower than that inferred from 
the list counts. Grandview is a good example of this problem in that the weighted sample 
estimates the number of students certified free by application is 856 compared to the provided 
initial list size of 1,156 (noting that the updated list also contained an even larger listing of free 
students, 1,443 (Table III.4). 

Finally, we note several problems related to identifying directly certified students. Staff at 
Creve Coeur, Oak Park and River Forest, and North Pekin and Marquette said they did not use 
direct certification at the time we were preparing the sample frame. However, this turned out to 
be incorrect, because in 2002 the state of Illinois began requiring all state districts to accept as 
directly certified all students who brought to school a letter indicating their eligibility based on 
food stamp or TANF receipt. As a result, we selected into the sample and interviewed some 
directly certified students who ended up being ineligible for the study. Because we could not 
replace these students with additional students who were free by application, this error produced 
smaller-than-planned samples for Creve Coeur and Oak Park.^^ 

We also misinterpreted the indicator of direct certification status in the data file provided by 
Jamestown City, and selected into the sample and interviewed 31 directly certified students. 
However, we detected this error during the data collection period and replaced these students 
with others certified free by application to maintain the desired sample size. 

*^No directly certified students were selected and interviewed in North Pekin and Marquette. 
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2. Sampling Steps 

The sampling steps varied from district to district, with a maximum of four sampling stages 
conducted, depending on the situation. The first step, when needed, was an interval sample 
selection of the students listed on the full enrollment list to reduce data entry. In five districts 
desiring to limit the release of the contact information to a sample of their students, we also 
conducted a secondary stratified sample selection process. Following these list reduction steps 
are the primary sampling steps for the selection of the student records, which we will refer to as 
the household selection step and the subsequent student- within-household selection. 

Table III.5 summarizes the sampling steps conducted for each district. Five districts — 
Montrose (PA), Creve Coeur (IL), North Pekin and Marquette Heights (IL), Williamson (TN), 
and Wilson (TN) — required that we select a sample of students so that they would need to 
provide MPR with contacting information only for the sample rather than for all students. For 
UFD comparison district 1 and UFD pilot district 9, we selected an interval sample from the 
entire full enrollment file to reduce the keying; in the other two districts, the full listing was 
keyed. From the full list/sample we then selected a stratified sample based on meal status using 
an allocation that roughly inflated the number of targeted interviews presented in Table III.l by 
40 percent to account for refusals. We sent this sample to the district, and they returned the 
address information to allow us to proceed to the household selection stage. 



*^We note that UFD comparison district 3, GV pilot district 3, UFD pilot district 6, and UFD 
comparison district 8 required passive consent, but they provided us with street addresses on the 
full enrollment list to eliminate the need for a pre-household sampling step. GV pilot district 3 
basically provided us with a l-in-5 sample of their students (365 students) for which they 
obtained passive consent and then returned to us the names and addresses for the consenting 
students (352 students) to merge to the sample file. 
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OVERVIEW OE SAMPLING STEPS CONDUCTED ON EACH DISTRICT 



Stage 3/4 




Households/Students 
Sample Size 




241 


261 


283 1 


291 1 


284 1 


091 


279 1 


291 1 


1 


291 1 


264 


231 1 


6ZZ 


251 


291 1 


233 


297 


170 


Stage 3 


Household Selection Method 


Paid 




PPS 


Equal 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


Equal 


PPS 


PPS 


Equal 


Equal 


Reduced 




Equal 


Equal 


Equal 


PPS 


PPS 


PPS 


Equal 


Equal 


PPS 


Equal 


Equal 


Equal 


PPS 


Equal 


Equal 


PPS 


Equal 


Equal 


Pree 




Equal 


Equal 


Equal 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


PPS 


Equal 


Equal 


PPS 


Equal 


Equal 


1 1 




1 n 






1 1 








1 1 










1 1 




1 1 


1 1 




1 1 


1 1 


1 1 


Stage 2 




Sample 

Size 






281 








NA 










NA 




291 


303 




NA 


320 


297 




Special 

Situation 






No addresses 
provided 








Passive 

consent 










Passive 

consent 




No addresses 
provided 


No addresses 
provided 




Passive 

consent 


No addresses 
provided 


No addresses 
provided 














































Stage 1 




Sample 

Size 




620 


448 


779 


7,829 


1,120 


3,677 


1,047 


SO 

O 

rn 

cn 


1,147 


1,600 


1,229 


589 


687 


099 


2,971 


1,862 


3,287 


1,107 




Interval 
(1 =NA) 








o 


- 




- 


IT) 


- 


- 


- 


CN 


fN 


- 


- 


- 


- 


CN 


- 






Pilot 

Comparison 

Status 




Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 


Oh 


u 






District 


Up-Front Documentation 


Blue Ridge, PA 


Montrose, PA 


1 East Stroudsburg, PA 


1 Easton, PA 


1 Pleasant Valley, PA 


Bangor, PA 


1 Stroudsburg, PA 


1 Pottsgrove, PA 


1 Maplewood, OH 


1 Newton Falls, OH 


Salem, OH 


1 Lisbon, OH 


Creve Coeur, IL 


North Pekin and Marquette Heights, IL 


1 Oak Park and Forest River, IL 


Valley View, IL 


Williamson County, TN 


Wilson County, TN 
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We wanted to interview each household only once in each phase of the survey, and about 
only one student. Because a fully student-based sample could yield more than one student 
selection from a household, we attempted to group the students by household. From the resultant 
listing, we selected a sample of households and then selected one child at random from each. 
Since we did not have any specific household linkage information from the school districts, we 
had to rely on the student’s address and parent’s name (when provided) to form the linkages. As 
a result, the household relationships created were not completely accurate, but this process 
greatly reduced the chance that we would end up with two students from the same household in 
the sample. During the interview, if two sample students were identified as members of the same 
household, the student that was discussed on first contact with the household became the student 
selected for the interview, and we inflated this student’s survey weight to compensate for this 
selection process. 

We created the household groups using either the Double-Take software or an exact match 
on parent’s last name and address. We decided which method to use depending on the time 
available to prepare the sample. As time permitted, we conducted a household identification 
process using the Double-Take*^ software package to parse the address information and to link 
records phonically if they had common addresses or parent last names. However, when we 
received the sampling lists shortly before the scheduled start of the field period for a district, we 
used exact matches on parent’s last name or address rather than the more time-consuming but 

1 8 

While this selection step was not completely random, the samples were originally sorted in 
a random replicate order, and only 51 such students were identified in the released sample of 
3,806. 

*^From Peoplesmith Software, 50 Cole Parkway, Suite 34, Scituate, MA 02066-1337. 
[www.peoplesmith.com] . 
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more accurate Double-Take methods. After we developed a set of tentative linkages using either 
method, we visually reviewed households that had three or more member students to confirm the 
linkage or to reject it if the information provided indicated that one or more of the linked 
students were not related. 

With the pseudo household linkages in place, for the next step we assigned a meal status 
designation to each household for stratification purposes. In most cases the household linkages 
identified students with the same meal status; however, for those that did not, we assigned a meal 
status to the household based on the highest level of benefit approved among the member 
students. We then also assigned to each household the number of student records associated with 
each and reduced the student listing to a single record per household for the sample selection 
process. We note that because in many districts we selected an initial interval sample of the full 
enrollment lists, few multiple student households were identified and the number of students 
listed reflected a sample count rather than the actual number present in the household. 

For the household selection step (Table III.5), for many districts we conducted an equal 
probability sample selection of the households in each free or reduced-price stratum and selected 
a probability-proportionate-to-size (PPS) sample of households in the paid stratum. For the free 
and reduced-price strata, the number of households available for sampling tended to be low, and 
we needed to select nearly all of the households, which precluded use of PPS sampling. In 
contrast, the base for the paid stratum was quite large, which allowed the use of a PPS selection 
process. When we were able to use a PPS approach, we used the number of students 
listed/sampled in these pseudo household groups as the measure of size. The use of a PPS 
sampling process for the households, coupled with the selection of one student at random from 
the selected households, produces a design that within each stratum gives the students the same 
probability of selection, which increases the precision in the survey estimates. In contrast with 
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an equal probability selection of households and selection of one student per household, selection 
probabilities vary among students as the result of differences in the number of students per 
household. 

As the final sampling and processing steps, we selected one student from each selected 
household and divided the sample of students into random replicates to allow for the possibility 
of making multiple sample releases. To select the students, we assigned a random-generated 
number between 0 and 1 to each and then sorted the students within each household based on the 
value of the random number. We then selected the first student from each household to complete 
the sample. In those districts with passive consent, we merged to the sample the passive consent 
outcomes and the contact information obtained. From the full sample we divided the sample into 
random replicates on a stratum basis. This process allowed us to generate a random subset of the 
sample to be released initially for interviewing and to supplement this release with another 
random subset as needed. With the short field period involved, we would be able to make only 
one sample release, and we used the replicates in each stratum to generate a sample equal to 
about 1.4 times the interviewing target (using the full sample if its size was smaller than this 
value) under the assumption that survey would yield a 70 percent response rate. One exception 
to a single sample-release occurred, in Jamestown (NY), where we released supplemental sample 
to replace some direct certification cases erroneously included in the initial release. 

C. MATHEMATICAL SUMMARY OF THE STUDENT SELECTION PROBABILITIES 

The value of the probability of selection of a student in this study can be mathematically 
expressed as product of multiple components associated with each stage of the student selection 
process as given in (1) 
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In (1), school districts are indexed by D, the meal status of the student is indexed by h, and 
the meal status assigned to the household by h’. The two SR terms reflect, respectively, the 
optional first- stage simple interval sampling methods to reduce the size of the list for keying and 
a second optional stratified systematic sampling process to reduce the sample further when the 
district did not want to furnish addresses for the full frame. The H term is associated with the 
household grouping and selection process using either a PPS or an equal probability selection 
process as discussed in Section B. In essence, the H term reflects the probability that a student 
selected during the initial sampling phases would remain in the sample during the household 
selection process. With an equal probability selection of households the measure of size 
(MOS) is equal to 1. The ST term reflects the selection of one of the students (previously 
selected) from each selected household. 



20 

MOS equals the number of students in the household, or if the initial list was sampled, the 
number of students sampled in the household at this point. 
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IV. SURVEY DATA COLLECTION 



A. SURVEY DESIGN 

The primary objectives of the Evaluation of National School Lunch Application/Verification 
Pilot Projects were addressed through a survey conducted with representative samples of 
students’ families in pilot and comparison SFAs. The survey was implemented in two parts: 



• Part 1. We used a short telephone interview to obtain data on the characteristics of 
households, as well as information on gross income and household size — information 
that allowed us to determine whether a household was ineligible for free or reduced- 
price school meals. Households determined to have incomes above 400 percent of 
the federal poverty level (FPL) during the telephone interview participated only in 
this first screening stage of the data collection. 

• Part 2. We conducted an in-person interview with households whose incomes were 
found, during the part 1 interview, either to be less than 400 percent of FPL or to be 
undeterminable as a result of lack of contact or refusal. We administered this second 
survey module to collect detailed income information and to review income 
documentation so that we could form an accurate independent estimate of the 
household’s eligibility for free or reduced-price meals. Households not interviewed 
by telephone in the first phase of data collection completed the part 1 interview at this 
time, and if their income based on part 1 was above 400 percent of FPL, they did not 
complete the detailed part 2 portion of the survey. 



We administered both parts of the survey using computer- assisted telephone interviewing 
(CATI) and computer-assisted personal interviewing (CAPI). We offered a two-tiered financial 
incentive to respondents at each stage of data collection. Sample members who completed part 1 
by telephone received $10 for that interview and an additional $20 for completing part 2 in 
person. Sample members who completed both parts of the survey in person received $20. 
Parents or guardians of students who were approved to receive free or reduced-price meals, as 
well as parents of students who were not approved for these meal discounts (either because they 
were not eligible or because they did not apply), were targeted for this survey effort. The sample 
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selection process is described in Chapter III. Figure IV. 1 shows the flow of cases from part 1 
(telephone interviewing) to part 2 (in-person interviewing) for a typical study district. 

B. INSTRUMENT DESIGN 

We developed two instruments to carry out the required data collection for this project. The 
part 1 instrument gathered basic information about the family and identified high-income 
households whose income eligibility for free or reduced-price meals could be established 
accurately through a brief series of simple questions. The part 2 instrument was developed to 
collect detailed income information. 

As specified in the RFP, the instruments were based on ones previously developed for WIC 
and other studies that examined application processes for means-tested programs. However, 
modifications in approach and content were necessary to meet the needs of the current 
evaluation. 

The part 1 interview was designed to take 15 minutes and was intended to be completed 
through either CATI or CAPI. The part 2 instrument was designed to take 30 minutes and to be 
completed using CAPI. 

Each instrument went through a rigorous, iterative development process leading up to and 
following the pretests, until the start of the main data collection effort. Internal reviews 
involving research and survey staff and external reviews involving the FNS Project Officer were 
ongoing. 

I. Part I Instrument 

The instrument developed for the part 1 survey collected basic demographic and household 
background, school lunch participation information, and gross income measures. Because 
collecting gross income information was key at this juncture, we asked about income in two 
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SUMMARY OF DATA COLLECTION IN A TYPICAL STUDY DISTRICT 
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ways. First, respondents were asked to select their household income from among ranges that 
were read. These income ranges allowed classifying the household’s income according to FPL. 
Respondents were then asked to give their total household income (before taxes and other 
deductions) for the previous month. 

Traditionally, these income questions are asked alternately, with the range question typically 
following the specific question for respondents who resist disclosing a specific income amount. 
The way income questions were asked in the part 1 survey was unusual in two ways: (1) both 
questions were mandatory, and (2) the range question was asked first. We chose this sequencing 
because we judged that asking the less invasive range question first would lead to fewer refusals. 
Also, since the opportunity to collect precise income information existed for all but those with 
incomes over 400 percent of FPL, it was important to establish good rapport and to avoid 
alienating respondents who would be asked to participate further. 

2. In-Home Survey Instrument 

The instrument developed for part 2 interviewing was designed to be administered in the 
homes of selected sample members, where they would have access to income documents. In a 
small number of cases, these surveys were conducted in places other than the home (such as 
workplaces, libraries, and other public locations) and even by telephone. 

For sample members who completed the part 1 interview by telephone, selected data from 
that interview were pre-loaded into the CAPI instrument. For example, the name of the sampled 
child and number of household members were noted on the file. As part of the in-home data 
collection, household members were listed, and their ages, relationship to the respondent, and 
genders were recorded. Financial relationships among household members and employment 
history was also ascertained for those over the age of 18. The remainder of this instrument 
collected detailed income information about sources for every household member over the age of 
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18, including earned and unearned income. We assumed that respondents who provided 
documentation of their current certification for either Temporary Assistance for Needy Families 
(TANF) or the Food Stamp Program automatically qualified for free school meals, so for them 
no additional income information was sought. The part 2 interview for this group was shorter 
than for those who did not receive TANF or food stamps. 

Respondents who reported income from wages and other sources were asked a series of 
questions about a comprehensive list of potential income sources. Questions included whether 
they or any household member received income from the source, the amount received during the 
past month, and the period covered by that income. After they answered these questions, they 
were asked to retrieve and refer to documents that showed the amount of income and to report 
this amount. Structuring the instrument in this way was important, because if respondents were 
reminded that documents would be requested at this point in the interview (the confidentiality 
acknowledgment noted this), the potential for underreporting income sources was thought to be 
higher. Respondents did, of course, have the option to decline to use the documents. 

3. Interviewer Observations 

The final part of the in-home survey was a set of six questions through which interviewers 
were asked to report on the interviewing experience. These questions asked about their 
perceived accuracy of responses, the respondent’s reaction to the request for documentation, and 
use of documentation to answer questions. 

C. SURVEY PRETESTS 

We conducted nine pretests of the telephone and in-person surveys prior to beginning data 
collection with the survey sample. The goals of the pretest were (1) to test the clarity, flow, and 
sequence of questions, (2) to determine respondent burden for both the telephone and in-person 
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portions of the survey, (3) to gauge respondent willingness to report income information, (4) to 
explore the incidence at which documentation is available, and (5) to determine respondent 
compliance with requests for verification documentation. 

Pretest respondents were diverse. They were from three New Jersey counties; their children 
attended schools in five different school districts; and they varied in terms of race/ethnicity, 
income, urbanization of their community, school lunch eligibility status, and household size. 

Procedures planned for the main study were replicated to the extent possible. The primary 
exception, necessitated by time constraints, was the use of paper- and-pencil methods to collect 
pretest data, unlike the computer-assisted approach used in the main survey. Pretests were also 
conducted by regular MPR project staff, rather than by on-call interviewers. Pretest respondents 
were paid $25 for completing both the telephone and the in-home portions of the survey. 

The pretest experience provided input for interviewer training and improvements to the data 
collection instruments. 

D. SURVEY IMPLEMENTATION 
I. Part I — Telephone Interviewing 

We attempted to interview all 3,685 sample members by telephone from MPR’s Princeton 
Survey Operations Center. The interviews were conducted by staff trained specifically for this 
project. Telephone interviewing began in mid-October and continued for about eight weeks, 
through early December. An average of five (5.3) calls was required to reach a sample member 
by telephone. Once the connection was made, the interview took an average of 15 minutes. 

The sample was released and worked in four waves for both stages of data collection. Cases 
were worked exclusively by telephone for 9 to 14 days, depending on the size of the sample, and 
then moved to part 2, field data collection. 
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Between each data collection stage there were two or three days during which contact 
sheets, checks, and other site specific materials were generated. Contact sheets contained 
appointment preference information collected during the telephone interview to help in-person 
interviewers schedule their work assignments. Separate $10 and $20 checks were generated for 
each sample member whose income was less than 400 percent of FPL, as well as for sample 
members for whom FPL was not determined in the Part 1 interview by telephone. We believe 
that hand-delivering the check for completing the part 1 telephone interview helped legitimize 
the in-person interviewer’s visit and contributed to increasing the response rate for the in-person 
interviews. We mailed $10 checks to sample members whose incomes were determined during 
the part 1 call to be above 400 percent of FPL. 

2. Part 2 Data Collection — In-Person Interviews 

The second survey stage, in-person data collection, began in late October and continued 
through mid-December. Seven data collection teams whose members were trained in Princeton, 
NJ in mid-October conducted the in-home interviews. Each team included 4 or 5 interviewers 
overseen by an on-site Team Leader, who was responsible for the assigning of cases and for the 
day-to-day on-site management of data collection activities. The Team Leader reported directly 
to MPR’s central office project staff, and interviewers reported to the Team Leader. Data 
collection staff conducted interviews at each district area for 7 to 11 days, depending on the size 
of the sample. 

As noted earlier, the in-person part 2 interviews were conducted using CAPI. Each 
interviewer was assigned a laptop that contained the part 2 survey instrument in CAPI format, as 
well as the CAPI version of the part I interview. In addition, each laptop contained SurveyTrak, 
a software program for electronic transmission of data. SurveyTrak automatically updated the 
status of cases completed in CAPI and, via telephone connection, sent MPR daily productivity 
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updates by site and interviewer. Reports this system generated allowed MPR project staff and 
Team Leaders to target problem sites and interviewers and take corrective action. 

For this data collection effort, MPR used a system whereby each member of an interviewing 
team was given access to all cases assigned to that team. While each team member was assigned 
specific cases to work, this arrangement provided flexibility in making mid-field changes in 
assignments as necessary. At the same time, limiting the laptops on which cases were loaded to 
site/team specific assignments kept interviewers from accessing cases erroneously. This flexible 
approach to assigning cases allowed team leaders to move cases among team members, as 
attrition occurred for a range of reasons, from illness to poor performance. 

As noted, the part 2 in-home interview collected more detailed information about the 
sources of income for all adult household members. During the in-home interview, respondents 
were asked whether documentation of each source of income was available and if so to retrieve 
it. They were then asked to refer to the documentation in reporting the amount of income from 
each source. Because of the sensitivity of this data collection, confidentiality was especially 
important. The next section describes the measures we took to ensure data confidentiality. 

While most interviews were conducted in sample members’ homes, some were conducted in 
other locations, such as libraries and work places, at the request of the sample member. A few 
part 2 interviews were also conducted by telephone. The average time for an in-home interview 
was about 30 minutes. 

3. Confidentiality Assurances 

Respondents to both stages of the study were assured that their participation in and answers 
to study questions would be treated confidentially. Confidentially was ensured by employing 
standards set by MPR for all data collection and by complying with the Privacy Act of 1974. 
Specifically, everyone hired by MPR is required to sign an oath of confidentiality as a condition 
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of employment. Physical safeguards, such as locked file cabinets, provide further protection for 
sample member contact information and survey data. 

An advance letter printed on FNS letterhead and signed by the FNS Project Officer provided 
written assurances to sample members that their participation was voluntary and confidential. 
This letter included a telephone number sample members could call to speak directly with the 
FNS Project Officer, as well as a toll-free number at which to call MPR with questions. The 
0MB clearance number was also displayed on the advance letter. These assurances were 
repeated as part of the introductory script to both stages of interviewing. 

To enhance the study’s legitimacy further, MPR solicited and received letters of support for 
the study from each participating pilot and comparison SFA or school district. These letters were 
mailed along with the USDA letter to each sample member in advance of any contact attempt. 
An information brochure was also included as part of the initial mailing. 

Finally, respondents to in-person interviews signed a Confidentiality Acknowledgment 
document (Figure IV. 2), which restated the purpose of the study and allowed the respondent to 
acknowledge that information regarding their participation, their payment, the survey content, 
and confidentiality was explained. The document, which was printed on 3-ply NCR paper, was 
also signed by the interviewer. The respondent retained a copy of the document, and the original 
and one copy were filed with MPR. 

Photo identification badges worn by in-person interviewers provided an additional source of 
reassurance for sample members. The ID badges were printed on MPR labels and contained the 
interviewer’ s name and MPR identification number, the name of the study, and the name of the 
sponsoring agency. 



59 




FIGURE IV.2 



NATIONAL SCHOOL LUNCH PROGRAM EVALUATION 
CONEIDENTIALITY ACKNOWLEDGEMENT 

I understand that the evaluation is being conducted hy Mathematica Policy Research, Inc. of 
Princeton, New Jersey for the U.S. Department of Agriculture (USD A). 

The purpose of the study is to help the USDA understand why some people apply to 
participate in the National School Lunch Program while others do not. It will also help the 
USDA understand difficulties people may have meeting program requirements or with the 
application process. 

During the interview I will he asked questions about my household, my children’s 
participation in the school lunch program, and about sources of income for my household. 
In order for the results of the study to be meaningful, I should, to the best of my ability, 
provide accurate responses to the questions. 

I understand that I have the right to refuse to answer any question I don’t want to answer and 
that my participation in this study is completely voluntary. 

I also understand that all of the information collected as part of this survey will be treated 
confidentially and used for research purposes only. My name will never appear in any 
reports and my answers will never be linked to my name. All of the answers given by 
people who participate in this survey will be combined and summarized for research 
purposes. 

I have been told that all researchers on this project have signed a Confidentiality Pledge 
which states that they cannot disclose any confidential information including names or 
identifying facts provided by individuals or families participating in this project. 

I also understand that both the interviewer and I will sign this agreement to show that the 
purpose of the study and its requirements have been explained to me and that I understand 
them. I will keep a copy of this agreement for my records. 

If I have any questions or concerns about my rights as a participant in this survey, I 
understand that I can call Sarah Lewis, toll free, at (800) 273-6813. 

I understand that I will receive $20 for completing the in-home survey. 

Printed Name of Respondent Printed Name of Interviewer 

Signature of Respondent Signature of Interviewer 

Date Date 
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4. Passive Consent 



Most districts participating in the study provided contact information for sampled families 
based on school district assurances from FNS and MPR that data would be held confidential and 
would not be used for any purpose other than the study. However, for four participating 
districts — Easton (PA), Dunkirk (NY), Maplewood (OH), and Valley View (IL) — a passive- 
consent process was used. In these districts, school staff sent the equivalent of the study’s 
advance letter and information packet to families selected for the sample in their district. In 
addition to the study information, the materials also instructed the families to call or indicate in 
writing if they wished to be excluded from the study. Families who made this request were then 
removed from the sample. In some cases, we received only ID numbers to identify the families 
before the consent process and received contact information only after the period for responding 
had expired. In other cases, the contact information was provided in advance, and we were told 
which names to remove from the sample. A total of 27 student families out of 661 released cases 
across the four districts opted out of the survey. 

E. SURVEY OUTCOMES 
1. Response Rates 

Response to the survey is summarized in Table IV. 1. A total of 3,806 students were selected 
for the sample, of which 3,685 were released for interviewing (eliminated were students directly 
certified, and one student in each households in which two students were selected). Of the 
sample released for interview, we determined that 163 were in fact ineligible for interview, 24 
did not provide their consent to be contacted, 478 did not complete the part 1 interview, and 
3,020 did complete the part 1 interview. Of these 3,020 respondents, 494 did not complete 
part 2. The key survey response rate is the proportion of the eligible sample who completed the 
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TABLE IV. 1 



SUMMARY OF SURVEY OUTCOMES 



Description 


Subtotals 


Students 


Starting Sample to Be Released for Interviewing 
Eliminated: 




3,806 


Directly certified in initial sample^ 


63 




Students found to be direct certified after sample prepared 


7 




Households with 2 students selected'’ 


51 




Final Sample Released for Interviewing 
Reduction: 




3,685 


Identified as ineligible during interview 


163 




Refused during passive consent 


24“ 




Survey nonrespondents to part 1 interview 


478 




Completed Part 1 




3,020 


Survey nonrespondents to part 2, completed part 1 


494 




Completed Part 2 




2,526 


Unweighted Overall Response Rate to Part 1)‘' 




86.3 


Unweighted Overall Response Rate to Part 2) 




72.2 



Tncludes 31 from Jamestown who were identified initially as directly certified but erroneously retained in 
the sample frame and 32 from various districts that were identified as directly certified after sampling (9 
from Oak Park, 3 from North Pekin, 13 from Creve Coeur, 5 from Dilworth, Glyndon, Felton, and 2 
from Wilson County). 

'’While we attempted to identify household relationships from last name and address information of the 
students and parents as provided on the districts enrollment frame provided, in some cases we were not 
successful, and we selected two students from 50 households that were not identified until interviewing 
was attempted. For these cases, we identified the household involved, eliminated the second student 
sampled, and doubled the original sampling weight of the remaining student to compensate for the 
duplicate household selections. 

‘’In total we had 27 refusals during passive consent for three of these, but we inadvertently completed an 
interview. 

‘'Eligibility is known for all sample released for interviewing less nonrespondents to part 1, or 3,207 cases 
(=3,685 - 478), of whom 3,044 are eligible (=3,207 - 163). The unweighted eligibility rate is .94917. 
Applying the eligibility rate to the full release sample of 3,685 gives 3,498 eligible released sample. The 
unweighted response rate is 86.3 percent (=3,020/3,498). The weighted response rate is calculated 
similarly but weights each observation by 1/base sampling rate. 
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part 1 interview. The unweighted part 1 response rate was 86 percent. This group formed the 
basis for estimating the program impacts reported for the evaluation.' 

Tables IV. 2 A and IV.2B provide additional detail on survey outcomes. Table IV. 2 A shows 
weighted and unweighted response rates to the part I interview, by district and meal price status. 
Table IV. 2B shows the same data for the part 2 interview. The response rates were similar 
across pilot types and within each pilot type across pilot and comparison groups. For example, 
the unweighted part 1 response rates for the full sample were 86 percent in the pilot sites and 
85 percent in the comparison sites in the Up-Front Documentation evaluation, and 89 and 
87 percent for the pilot and comparison sites, respectively, in the Graduated Verification 
evaluation. Weighted response rates tended to be slightly lower than the unweighted rates in the 
Up-Front Documentation evaluation but were nearly identical overall in the Graduated 
Verification evaluation. 

Response rates vary considerably by meal status and district. For example, the part 1 
response rate for free certified students varies by site from 74 percent to 100 percent, for 
reduced-price students from 71 to 100, and for paid students from 73 to 100. The part 1 response 
rates for the full sample by site also vary considerably: from 79 to 98 percent. 

2. Survey Validations 

We attempted to validate all interviews completed in person. We sent a thank-you letter to 
each respondent, asking them to complete and return an addressed, stamped postcard to MPR 
indicating whether and how they completed the interview. Exhibit IV. 1 shows the content of the 



*We used the sample who completed the part 2 interview to examine the sensitivity of key 
estimates to the decision to base the analysis on the part 1 sample. See Chapter IX. 
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validation postcard. Letters and postcards were prepared in both English and Spanish, and 
respondents received the validation packet in the language they used to complete the survey. 

Thirty-five percent (812) of part 2 survey completers returned the validation postcards to 
MPR. One hundred percent of those returning the postcards confirmed their participation in the 
study. The overwhelming majority had very positive remarks about the survey and the 
interviewers. 
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PART 1 SURVEY RESPONSE RATES, BY DISTRICT AND MEAL PRICE STATUS 
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PART 2 SURVEY RESPONSE RATES, BY DISTRICT AND MEAL PRICE STATUS 
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EXHIBIT IV. 1 



VALIDATION POSTCARD (SIDE 2) 

1. Did an interviewer compiete a survey about the schooi iunch appiication process in your 
home? 

1 □ YES 0 □ NO 

2. Did you compiete a survey by teiephone? i □ YES o □ NO 

3. Was the interviewer poiite and professionai? 

1 □ YES 0 □ NO 

4. How much were you paid for your participation? $ 

5. Piease provide any comments about the survey or interview process beiow. 



THANK YOU FOR COMPLETING THIS POSTCARD. PLEASE DROP IT IN THE MAIL. 
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V. COMPARING PLANNED AND ACTUAL SAMPLE SIZES AND PRECISION 



In this chapter, we examine the extent to which the sampling and data collection procedures 
produced the planned sample sizes and precision levels. 

Table V.l shows, by district and meal price status, the target number of interviews, the 
number of part 1 completed interviews, and the number of part 2 completed interviews. Overall, 
the total of 3,020 part 1 completed interviews exceeded the planning target of 2,718.^ Within the 
Up-Front Documentation districts, total completed part 1 interviews for the free and paid strata 
exceeded the targets in pilot and comparison sites, while the number of completed reduced-price 
interviews was below target in the pilot sites, slightly above target in the comparison sites, and 
below target overall. Considerable variation is apparent across districts in whether the targets 
were met, especially for reduced price (where 10 districts were over the targets and 12 were 
under) but also for free (where 6 of 18 were under). The Graduated Verification pilot and 
comparison districts exhibit very similar patterns. 

The precision levels achieved were broadly consistent with expectations, although our 
planning assumptions in some domains proved incorrect. Tables V.2A shows, for the Up-Front 
Documentation pilots, the planned and actual number of sample points per analytic group, the 
population proportion for the comparison group, and the minimum difference detectable using 80 
percent power and a two-sided 95 percent hypothesis test. The analytic groups are numbered for 
reference in the discussion. 

The minimum detectable difference is the smallest true difference for which we will reject 
the hypothesis that the proportions are the same, using a 95 percent two-tailed test in 80 percent 
of the identically selected random samples from the population. A smaller MDD reflects greater 

'chapter III describes how the planning target was established. 
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PLANNED VS. REALIZED OUTCOMES AND PRECISION LEVELS IN 
UP-FRONT DOCUMENTATION DISTRICTS 
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sample precision. The MDD is calculated as the sampling error of the difference in the two 
proportions we are comparing times 2.806 (which reflects the two-tailed 95 percent hypothesis 
test and 80 percent power). For our planning estimates the sampling error of the difference was 
calculated as: square root of [(p*(l-p)*deffl)/ nl -i- (p *(l-p)*deff2) /u2] where p is the average 
value of the two proportions being compared and nl and u2 are the sample sizes of the pilot and 
comparison districts. Deffl and deff2 are design effects associated with these two groups that 
resulted from a differential sample allocation. The sample allocation model described in Chapter 
ITT was used to estimate the deff terms by district based on the planned allocation of the sample. 
These terms were then averaged for the four groups created by crossing pilot type and pilot vs. 
comparison status for each of the analytic domains. 

We had mixed success in predicting the number of sample points per analytic group and the 
relevant proportion in the comparison group. For example, in the Up-Front Documentation 
districts, we overestimated the population proportions for groups 2*, 3, and 4 (students in 
households with income above the eligibility cut-offs) and underestimated the sample sizes 
(offsetting deviations from plan). For groups 5 and 6* we underestimated the proportions 
(increasing the variance since the actual proportion was closer to .5 than the planning estimate) 
and underestimated the sample sizes (again offsetting deviations from plan). For analytic group 
1, the realized sample was 13 percent larger than planned, and the population proportion was 
approximately the same as our planning assumption. Yet the MDD was nearly 25 percent larger 
than our planning estimate. This occurred because we incorrectly had assumed a simple 

2 

The sample size and allocation among strata were determined on the basis of analysis group 
and outcomes #1, #2, and #6. These key analysis group and outcomes are designated with a * 
next to the number. We devoted considerable effort to estimating the corresponding proportions 
accurately. Less effort was invested in estimating the proportions for #3, #4, and #5. Indeed we 
believe the estimate of the population proportion of .30 for group #3 was inconsistent with the 
estimate of .10 for group #2. 
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comparison of proportions in our planning, which did not properly account for the complex, 
nonlinear estimation of the accuracy outcomes. 

Table V.2B presents similar data for the Graduated Verification pilot and comparison 
districts. Our predictions of sample sizes and proportions were somewhat more accurate, but not 
much. We underestimated the number of sample points in groups 2*, 3, and 4 and overestimated 
the numbers in groups 5 and 6*. Our projections of the proportions were fairly accurate, except 
for group 3 (students above 130 FPL), which we overestimated substantially. Analysis of the 
relationship of the standard errors of the proportions to the MDDs indicates that stratification 
improved precision in group 4 (income over 185 FPL), but design effects reduced it in group 6* 
(income under 185 FPL). Comparing the ratios of standard errors with the ratios of MDDs 
suggests that for groups 3, 4, and 5 small precision losses due to smaller than expected samples 
may have been offset by modest gains from stratification. 

The precision achieved appears to be roughly in line with our planning assumptions. The 
differences in proportions (or percentage points) detectable with our actual design were smaller 
than the detectable differences in our original sample planning for most groups. Two exceptions 
to this general finding that the minimum detectable differences from our actual sample were 
smaller than our planning estimates are for households with income below 130 percent FPL and 
for households with incomes below 185 percent FPL. For these groups the MDDs were larger. 
Relatively large within-group differences in weights produced design effects, which caused this 
pattern. 



-5 

As noted in the note to Table V.2B, we believe our estimate for group 3 was inconsistent 
with the estimate for group 2*. 
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PLANNED VS. REALIZED OUTCOMES AND PRECISION LEVELS IN 
GRADUATED VERIFICATION DISTRICTS 
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VI. UPDATING MEAL PRICE STATUS 



For analysis purposes, our goal was to classify all sampled students according to their meal 
price status as of October 31, 2002. However, as described in Chapter III, many of the meal 
price status lists used for sample selection did not reflect each student’s status on that date. For 
some districts, the sampling list reflected meal price status at the end of the previous school year. 
In others, the list was generated before the 30th school day of the 2002-2003 school year (when 
students who had been certified for the prior year but who had not submitted applications and 
been approved for the current school year would still have been certified). Furthermore, our use 
of electronic matching procedures for sampling purposes could have led to inaccurate 
classifications of some students. To determine meal price status as of October 31, 2002, we 
obtained new lists for those districts that had initially provided a listing generated before the 30th 
day of the current school year, and we matched students on these lists with the entire sample of 
students selected for interviews. Also, as a check on the accuracy of the classifications 
determined from the electronic process used for sampling, we visually matched the initial lists 
received from districts that provided such lists after the 30th school day in fall 2002. We used 
this meal price status as of October 31, 2002, for preparing the survey weights and the data 
analysis. 

Table VI. 1 shows for each district whether we obtained an updated list and whether we 
visually matched the list to our sample list (column labeled “list status”).^ As with the initial 
lists, most updated lists were received in hard-copy format and were entered to electronic files. 



*We did not conduct a rematch with the direct certification lists, which would not have been 
updated. In districts in which the free and reduced-price listing also identified the direct- 
certification cases, we compared these cases to our sample and, if we found a match, made the 
sample member’s final status “ineligible for the study.” 



75 




UPDATED LIST MEAL STATUS COUNTS AND TRANSITIONS 









Percentage 
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T = Updated meal price status listing was obtained, 2 = No updated meal price status listing was obtained; the original listing was manually re-matched to the enrollment list, 3 = Original list and match was 
judged accurate. 





Two clerks independently visually matched the updated meal price status listing to a list of 
all sample released cases for interviewing, and marked on the sample listing the meal price status 
of each student (free, reduced-price, or not on the free or reduced-price listing and therefore 
paid). The updated free and reduced-price listings and the released sample of students were 
preprinted in order by last name and first name for the review process. When available, the 
printed versions provided the grade of the student, parent names, and addresses. The clerks 
matched names and considered a match as valid if (1) the grades on the released sample list and 
the updated free and reduced-price lists were within one year of each other, or (2) the grade was 
missing on either source. If any potential matches were questionable, the clerk marked the 
updated status as unknown. After each clerk matched the new lists to the sample, we compared 
the updated meal price status information, and printed out all cases that contained any 
disagreements in the status values or to which either clerk had assigned a status of unknown. 
The clerks and a supervisor reviewed the cases with disagreements and unknown statuses and 
assigned a final status to the sample record. 

Table VI. 1 shows the number of records by meal price status at sampling process and by 
their meal price status on October 31, 2002, based on the updated list and clerical review. For 
example, at Blue Ridge, PA, 26 cases were classified free at sampling and their status on October 
31 was free, 2 were free at sampling and their status on October 31 was reduced-price, and 2 
were free at sampling and their status on October 31 was paid. The final column shows the 
percentage of cases in which the original meal price status classification is the same as the final 
classification. Overall, the agreement rate was 88.4 percent. It ranged from 75.1 percent to 
100 percent. 

The large number of status transitions underscores the importance of having acquired the 
updated lists and manually reviewed meal price status. Among the 15 districts for which we 
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both acquired a new list and conducted a manual match, 84 percent of cases remained 
unchanged, and the percentage unchanged varied from a low of 75 percent to a high of 
100 percent. Among five of the six districts where we did not acquire a new list but did conduct 
a manual review of the initial list, the percentage of cases unchanged was at least 96 percent. 
However, in one of these six districts, the percentage with unchanged status was just 85 percent. 
The impact of these adjustments on the population estimates is discussed in conjunction with the 
weights in Chapter VII. 
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VII. SURVEY WEIGHTING PROCEDURES AND POPUUATION ESTIMATES 



A. OBJECTIVES AND OVERVIEW OF THE WEIGHT CALCUUATIONS 

We developed survey weights that account for the complex nature of the sample design and 
the multi-stage data collection. The weights are based on sample selection probabilities and 
adjusted for nonresponse of some sampled households to the study interview. The survey 
weights are projection weights: when they are applied to the data for the sample, they produce 
estimated totals similar to the totals that would be obtained from a complete enumeration of the 
study population (defined as all students enrolled in districts in the study except those directly 
certified for free meals). The weights also align the characteristics of the sample to match those 
of the study population. Therefore, they provide unbiased estimates of population means, 
proportions, and totals for the various analytic groups. 

Nonresponse adjustments are an important element of the weights. The data collected for 
this study are subject to unit nonresponse and item nonresponse. Unit nonresponse occurs when 
none of the survey items are collected because the sampled student/parent refuses directly to 
complete the interview or is unreachable during the survey period. Item nonresponse occurs 
when the person participates in the survey but does not answer all the questions. The 
nonresponse components of the weights adjust for unit nonresponse to both the part 1 and part 2 
interviews. Accordingly, we prepared two sets of weights: one supports the analysis of the 
sample who completed the part 1 interview, the other supports analysis of the sample who 
completed the part 2 interview.' 

'item nonresponse to the part 2 data was handled by the use of data imputation as discussed 
in Chapter VIII. 
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1. Overview 



The population represented by our sample ineludes all students who were enrolled at the 
start of the 2002-2003 sehool year in the pilot and comparison districts and who were not 
directly certified for free meals. The weights for respondents to part 1 consist of the product of 
the first three components listed below. The weights for respondents to part 2 consist of the 
product of all four components: 



1. A base projection weight for each student in the eligible released sample (n=3,685), 
which is the inverse of the student’s probability of selection. It accounts for 
differences in the sample selection rates. This component actually consists of four 
sub-components associated with the four potential stages of sample selection (see 
expression [1] in Chapter III.C for a discussion and mathematical summary of these 
factors). 

2. An eligibility rate adjustment deflates the sum of the base weights, and the 
corresponding estimate of the study population, to account for the fact that some 
sampled students were found to be ineligible for the study during the interviewing 
process. 

3. A part 1 nonresponse adjustment to account for differences between the part 1 
respondents and the released sample of students. This component was based on 
information about the sample members obtained from the sampling frame, and used 
the CATI interviewing outcomes to further subdivide the part 1 completes into CATI 
respondents and nonrespondents in forming the adjustments. 

4. A part 2 nonresponse adjustment to account for differences between the part 1 
respondents who did and did not complete the part 2 interview. This component was 
based on information in the part 1 interview. 



To finalize the weights for the survey completes, we realigned the weighted distributions to the 
estimated population totals. 

These weights provide unbiased estimates of means, proportions, and totals for the target 
population if two assumptions hold true. First, we assume that each district provided a complete 
list of their enrolled students at the start of the 2002-2003 school year. From this list we selected 
a scientifically based, stratified sample of students with known probabilities of selection. While 
the data used to stratify the sample were subject to error, the base weights project to the total 
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population of students on the lists. Second, we assume that the list updating and matching 
process described in Chapter VI enabled us to classify accurately each sampled student’s meal 
price status as of October 2002. If these two assumptions are correct, the weighted sample count 
is an unbiased estimate of the population of non-directly-certified students in each district by 
meal status (free on the basis of an application, reduced-price, and paid). 

2. Illustrative Calculations of the Weights for One District 

To illustrate the computations, we use data from tables III.l, III.4, V.l, and VI. 1, which 
provide counts for the associated sampling, data collection, and data processing or operational 
procedures. We will use as our example the Blue Ridge school district (Table VII. 1 displays the 
data). We received a full enrollment list from this district that contained 1,240 students. Based 
on our initial processing, we stratified the students into three groups — 212 certified free based on 

application, 95 certified reduced-price and 933 balance paid (this district did not have any 

2 

students directly certified for free meals). 

We ultimately released a stratified sample of 132 cases, which included 30 free students, 
16 reduced and 86 paid. The base weight component accounts for the multistage sampling 
process used. Dividing the number of free students (212) by the number selected (30) yields 
7.06, which is close to the average base weight of 6.93. Similarly, the number of reduced-price 
students divided by the number selected is 5.94, close to the average weight for the group of 
4.50. The number of paid students divided by the number selected is 10.84, which is close to the 



2 

In this district we initially selected a l-in-2 sample from the initial enrollment list, which 
yielded 107 free, 46 reduced-price, and 467 paid students. 



81 




TABLE VII. 1 



EXAMPLE COMPUTATION OP POPULATION ESTIMATES POR BLUE RIDGE, PA, SCHOOL DISTRICT 













Updated Meal Price Status 


Adjusted Population 
Estimates” 


Original 

Stratification 


Sampling 

Prame 

Count* 


Cases 

Sampled 

and 

Released’’ 


Average 

Base 

Weight 


Original 
Weighted 
Estimates of 
Population 
Counts” 


Outcome 


Cases'* 


Weighted 

Count 


Free 


Reduced 


Paid 


Tree 


212 


30 


6.93 


208 


Free 


26 


180 


180 
















Reduced 


2 


14 




14 














Paid 


2 


14 






14 


Reduced 


95 


16 


4.5 


72 


Free 


4 


18 


18 
















Reduced 


6 


27 




27 














Paid 


6 


27 






27 


Paid 


933 


86 


11 


946 


Free 


12 


132 


132 
















Reduced 


3 


33 




33 














Paid 


71 


781 






781 


Total 


1,240 


132 




1,226 




130 


1,226 


330 


74 


822 



“Source is Table III.4. 

'’Source is Table III.l. 

“Weighted estimates of population counts differ from sampling frame counts due to sampling from the enrollment list. 

‘‘Source is Table VI. 1. 

“The totals free, reduced-price, and paid shown for Blue Ridge in Table VII.2 differ from the figures shown here because we found one 
case that was a duplicate of a second interview with the same household (which was removed) and because there was variation of 
weights. 
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average weight of 1 1 . When the weights are summed for the sample in each group, these sums 
are close to the original sampling frame counts of 212, 95, and 933. However, because of the 
variation from sampling from the initial enrollment list, the sums of weights are 208, 72, and 
946, for a total of 1,226. 

The sum of base weights provides estimates of the population sizes that are subject both to 
sampling variation and to error in the classifications according to meal price status. To correct 
for the error in classification, we conducted a matching process of the sample cases to updated 
lists of students approved for free and reduced-price meals, as discussed in Chapter VI. As 
shown in Table VI. 1 of the 132 cases in Blue Ridge, about 78 percent had the same classification 
according to the initial list and the updated list. Of the 30 free cases, 26 remained as free, 
2 changed to reduced price, and 2 changed to paid. Similarly, among the 16 initially reduced 
price, 4 changed to free, 6 stayed reduced, and 6 changed to paid. Of the 86 initially classified as 
paid, 12 changed to free, 3 changed to reduced-price, and 71 remained paid. Applying the base 
weight to the new meal price classification produces a revised estimate of the populations by 
meal status group. With these changes in meal price status, summing the weights gives a final 
population estimate of 330 free approved, 74 reduced-price approved, and 822 paid, for a total of 
1,226. 

Based on subsequent review of the data collection outcomes we found that one of the 
initially free cases that stayed free was actually the second interview for household in which the 
second child had changed to paid status. We determined that the record which changed to paid 

The sums of weights are not the same as the frame values because they are subject to 
sampling variability. Given the small size of the sample of reduced-price students in this district, 
the differences between the sums of the weights and the frame values were larger for this group. 
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status contained the correct information on the household. Therefore, we doubled the weight on 
that paid status case and eliminated the free case to produce the final population estimates that 
are shown in Table VII.2 — 321 free, 78 reduced-price, and 833 paid, for a total of 1,232. 

The nonresponse adjustments — components three and four — use a ratio adjustment that 
inflates the sum of the base weights for completed interviews for part 1 or 2 to the sum of the 
base weights for the full sample. In our example above, the sum of the base weights for the 132 
released sample cases in Blue Ridge is 1,232, and we had 125 that completed part 1 of the survey 
and 108 that completed part 2 (see Table V.l). The sum of base weights for the 125 part 1 
completes is 1,152, which produces an average adjustment of 1,232 divided by 1,152, or 1.07. 
As discussed in Section C below, the part 1 nonresponse adjustments were actually calculated on 
a cell basis using the analytic meal price status, education level, and CATI outcomes status to 
reduce the potential for bias in the survey estimates. 

To finalize the weights, we aligned the sum of the weights (without applying the eligibility 
adjustment) for the part 1 and part 2 completes (using the product of components [1] one and 
three, or [2] one, three, and four, respectively) to match the population totals estimated from the 
full released sample (3,685) presented in Table VII.2. We then applied the second component, 
the eligibility adjustment, to deflate the sum of these weights to account for the fact that a small 
percentage of our released sample was found to be ineligible for the study during interviewing 
(for example, because the child was no longer enrolled in the study school district). In Blue 
Ridge, we determined eligibility for the survey for 129 of the 132 sampled cases, and found that 
96.2 percent were eligible for the survey, to produce an estimate of the population as it existed at 
data collection of 1,185. 
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ESTIMATED POPULATION DISTRIBUTION OF MEAL STATUS, BY DISTRICT 
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B. POPULATION ESTIMATES 



As discussed, using the final meal price status, we estimated district student populations by 
meal price status at the time of sampling and interviewing. Table VII.2 presents two sets of 
estimates. The first is based on the updated outcomes for the full released sample of 3,685 
students weighted by the baseline weight. The second, which is limited to the cases with 
completed part 1 interviews (n=3,020), accounts for the observed ineligibility of the sampled 
cases as determined from the interviewing process. These counts are weighted by the product of 
the baseline weight (component one), a nonresponse adjustment factor to compensate for survey 
nonresponse in part 1 (component three), and an eligibility adjustment (component two), to 
account for the fact that some of the students no longer attended school in the districts at the time 
of interviewing. 

As a check on the sampling process, it is useful to compare administrative counts of relevant 
population parameters with estimates of the corresponding data items based on the weighted 
study sample. The left portion of Table IV. 3 shows administrative counts of selected data items, 
measured as of October 31, 2002, for schools participating in the study: (1) total number of 
students enrolled, (2) number of students approved for free meals (including students directly 
certified), (3) number of students approved for free meals by direct certification, (4) number of 
students approved for free meals by application (excluding directly certified), (5) number 
approved for reduced-price meals. These data were furnished to FNS either directly by the 
district or by the state agency administering the NSLP in the state in which the district is located. 
Data on the number of students directly certified in the comparison districts were not available to 
us from this source. 

The table presents two comparisons: (1) a comparison of total number of students enrolled 
as measured by the administrative data and as measured by the weighted study sample, and (2) a 
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comparison of the percentage of all students enrolled who are approved for free or reduced-price 
meals (including those directly certified under both approaches). Details of the calculations of 
the sample based estimates are presented in the footnotes to Table VII.3. 

Column 8 presents the difference between the administrative data total enrollment and the 
Total Sample Population by Method 1 (all students in the district except those directly certified 
for free meals) plus the number directly certified expressed as a percentage of the total 
enrollment as reported by the district in administrative reports. We anticipate some differences 
in these two estimates of total enrollment because of the differences between the date the 
enrollment list for sampling was created and the count for October 31. In 15 districts, the 
absolute value of the percentage difference is less than 5 percent, in 7 districts it is 5 to 7 percent, 
and in 2 districts it exceeds 7 percent. In Creve Coeur, where the administrative-based estimate 
is 25 percent greater than the sample-based estimate, the enrollment list contained a smaller 
number of names than the administrative data (687 compared to 750); subsequent processing 
also identified students to further reduce the estimate of total enrollment. In Montrose, 
administrative data show 1 1 percent more students than the sample-based estimate. Overall, the 
differences are in line with our expectations. 

To examine the extent to which the administrative data and sample-based estimates provide 
estimates of the percentage of students who are certified for free or reduced-price meals, we 
calculated this percentage for the sample (column 9) and for the administrative data (column 10). 
The difference in percentages expressed as a percentage of the administrative based estimate is 
shown in column 11. Finally, column 12 presents an estimated standard error of the sample- 
based estimate. The difference is less than one standard error of the sample proportion in 14 of 
24 districts, between one and two standard errors in 9 districts, and greater than two standard 
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ADMINISTRATIVE DATA REPORTED BY DISTRICTS AND COMPARISONS WITH SAMPLE-BASED ESTIMATES 
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na = administrative counts not available. 
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errors in 1 district (and therefore statistically significantly different from zero). Again, the 
magnitude and standard errors of these differences indicate that the differences are within the 
range of what we would expect because of sampling variation. 

C. WEIGHTING METHODOLOGY 

This section provides details on the computation of each of the weighting components, 
which the reader may wish to skip after reviewing the overview presented in sections A and B. 

I. Component One: Sample Selection 

The first component is based primarily on the sample selection probabilities. It is the 
product of (1) the inverse probability of selecting the student, which was a four-component 
probability; and (2) a ratio adjustment factor to account for the fact that only a random subset of 
the total sample selected was in fact released. Initially, we selected a relatively large sample and 
divided it into random replicates. We then released initially a random subset of the sample that 
would yield the desired number of interviews if we achieved a 70 percent response rate. We 
planned that, if the actual response rate was below 70 percent, we could supplement the initially 
released sample with some of the unused replicates to achieve the targeted number of interviews. 
However, the high response rate of 86.3 percent for the part 1 completes rendered the additional 
replicates unnecessary. Because only a random subset of the full sample was released, we 
computed the ratio adjustment factor based on district and meal price status (at time of sampling) 
by setting it equal to the number of replicates released divided by the total number created."^ 



^ For a few districts, we set this ratio equal to the number of sampled cases divided by the 
number released rather than the number of replicates, since the samples in some districts were 
found to contain directly certified students, as discussed in Chapter III. 
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2. Component Two: Eligibility Rate Adjustments 

To develop the second and third weight components, we assigned each sampled case to one 
of nine weighting groups based on their CATI and CAPI interviewing outcomes and key data 
items, as follows^: 

1. Part 1 and 2 complete (part 1 income less than 400 percent of FPL; part 2 income 
less than, equal to, or greater than 400 percent of FPL) 

2. Part 1 complete and income at least 400 percent of FPL; part 2 not applicable 

3. Part 1 complete and income less than 400 percent of FPL, part 2 nonrespondent 

4. Found to be ineligible at CATI or CAPI interviewing stage 

5. Refused during district passive consent process 

6. No interview completed, part 1 contacted, part 2 contacted 

7. No interview completed, part 1 contacted, part 2 not contacted 

8. No interview completed, part 1 not contacted, part 2 contacted 

9. No interview completed, part 1 not contacted, part 2 not contacted 

We defined groups 1 to 3 as part 1 completes, groups 1 and 2 as part 2 completes, and 
groups 1 to 5 as having a known eligibility status for purposes of determining eligibility rates. 

Using this classification, we calculated the eligibility rate adjustment component to 
compensate for the observed ineligibility of some sampled cases. Initially, we formed 75 cells 
by combining district (25 groups) and meal price status (3 groups) to account for possible 
differences along these domains. Within each cell, we computed a ratio adjustment equal to the 
number of students known to be eligible (weighting groups 1 to 3 and 5 above) as weighted by 
the first weight component divided by the number of students with a known eligibility status 
(groups 1 through 5). To avoid creating an adjustment factor for a cell with less than 20 sampled 

^Complete status is assigned to a questionnaire for which the income data were sufficiently 
complete to use in the corresponding part of the data analysis. 
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cases, we collapsed the original cells by combining the free and reduced-price groups into a 
single category as needed to obtain at least 20 sampled cases in the final adjustment cell. We 
combined free and reduced-price status in 12 of the 25 districts to produce a total of 63 cells for 
the eligibility adjustment process. The adjustment factors ranged from .812 to 1.00, with a mean 
of 0.96. Overall, we estimated the eligibility rate of the sample to be 95.57 percent (weighted by 
the component 1 sample probability-based weight). 

3. Component Three: Part 1 Survey Nonresponse 
a. The Data Collection Steps as They Relate to Nonresponse 

To take best advantage of the information obtained at each stage of data collection for 
reducing the potential for survey nonresponse bias, the weighting components discussed in this 
and the following section recognize that the data collection methods do not directly overlap with 
the part 1 and part 2 analytic groups. In particular, the part 1 completed interviews included 
some cases that were done by telephone and some that could not be thus interviewed but 
completed the interview in person, a factor that we anticipated would be related to other 
socioeconomic characteristics of the household important for consideration in the nonresponse 
adjustments. 

For purposes of calculating weights, the data collection process encompassed three stages 
and used two methods. First, we attempted to interview each sampled student’s household by 
telephone using CATI methods. This interview obtained basic information to classify 
households into two groups, those with very high income (more than 400 percent of the federal 
poverty level [FPL]) and those with income below this threshold. This initial stage produced 
three outcomes: (1) nonresponse or non-complete with unknown eligibility status, (2) ineligible 
status, if the student was found to be a member of a group home or was no longer attending 
school in the district (for which interviewing terminated), or (3) completion of the part 1 
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interview. If the household’s ineome was 400 pereent of FPL or greater, no additional data 
eolleetion was required. If income was less, we attempted a part 2 in-person interview. 

As stage 2, if the household had not completed the CATI interview (or income status had not 
been determined), we attempted to obtain the missing part 1 data in the in-person interview. This 
interview resulted (as in the CATI interviewing) in either the successful collection of the data (a 
determination that household income was at, over, or under 400 percent of FPL) or a 
nonrespondent outcome. 

As stage three, if the student’s household income was determined to be less than 400 percent 
of FPL in the part 1 interview, the interviewer administered the part 2 detailed income battery. 
For the CATI completes, because the part 1 data had already been collected by telephone, the in- 
person interview began at stage three to obtain the part 2 data. 

b. Methods 

In preparing the nonresponse adjustments, our goal was to identify and use all available 
characteristics that were related to key survey outcomes and exhibited different patterns of 
nonresponse to the part 1 interview. Meal price status, grade, and district are the only student 
characteristics available. In addition, we also examined as discussed above the outcomes of the 
CATI data collection process as a proxy for potentially important but unavailable characteristics. 
Failing to make contact with a sampled household during the CATI stage may indicate a higher 
likelihood that the household does not have a telephone or experiences lapses in phone coverage. 
Lack of phone coverage in turn tends to be related to socioeconomic standing. Thus, lack of 
contact by telephone may distinguish the most severely economically disadvantaged households 
from a broader low-income group. By combining contact or lack of contact by telephone in the 
CATI phase with the sampling frame characteristic of free, reduced, and paid meal status and 
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grade level, we expected to be able to reduce the potential for bias in the survey relative to that 
provided by using the meal status and grade alone. 

We prepared the part 1 nonresponse adjustment using a weighting class approach similar to 
that for the eligibility rate adjustment. The sampled cases are assigned to cells that have 
homogeneous response patterns so that, within a cell, survey nonresponse is basically random. 
Within each cell, a simple ratio adjustment is calculated, which equals the weighted count of all 
cases in the sample (weighed by the base weights) divided by the weighted count of the part 1 
respondents (groups 1 to 3). For the final part 1 nonresponse factor, this is multiplied by the 
second component (the eligibility rate adjustment) to deflate the count in the numerator of this 
adjustment for observed ineligibility. 

As shown in Table VII.4, nonresponse patterns for the part 1 completion status showed 
some minor variation in the unweighted cooperation rates by meal price status, with the free and 
the paid categories having somewhat lower rates than the reduced-price students. High school 
students showed a lower completion rate than the other groups. The largest difference was based 
on CATI contact status: cases not contacted in CATI had much lower response rates to the part 1 
interview. This same pattern is evident in each meal price status group. 

In light of these findings, we created an initial set of adjustment cells by combining district 
membership, grade range (high school vs. other), and CATI contact status, which produced 253 
cells with 14.5 sample points per cell on average (3,685 released cases divided by 253 cells). We 
note that selecting a minimum cell size must balance two competing objectives: (1) the need to 
minimize the bias in the estimates (which argues for as many cells as possible), and (2) the need 
to minimize differences across cells due to sampling variation rather than real differences (which 
argues for fewer cells with more cases). We decided to collapse the categories to attain a 
minimum of 20 sample members per cell, subject to two constraints. First, because meal price 
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“Part 1 cooperation rate is number of completed interviews divided by number of sample released. 

'’Significance tests indicate whether the completion rates of the alternative categories within each student characteristic are significantly different from each other at the 0.05 (*) or 0.01 (**) level. 
‘’CATI contact status indicates whether any member of the student’s household was ever reached by telephone in the attempt to complete the part I survey. 




status was a primary analytical domain, we did not collapse across the final meal price 
categories. Second, by not collapsing across districts, we established a minimum of 75 cells (25 
districts by 3 meal price status groups). 

By district and meal price status, we combined cells across the grade range (collapsing high 
school and other to a single cell for all grades), or CATI contact status (collapsing from not 
contacted vs. contacted to a single cell of all cases), or both grade and CATI contact status, as 
necessary, to obtain a minimum cell size of 20, subject to the two constraints above. Because 
CATI status was more highly correlated with completion status, we collapsed the grade ranges 
first. If collapsing on grade did not meet the cell size requirements, we collapsed on CATI 
contact status (leaving the two grade range cells intact). If neither option provided a minimum 
cell size of 20, we then created a single cell based on district and meal price status membership. 

This process produced a total of 91 adjustment cells for the 3,020 part 1 completed 
interviews. These 13 cells have fewer than 20 members, of which 11 were for the reduced-price 
group. The minimum cell size overall is 7, and the minimum sample size across the 66 free or 
paid meal status cells is 18. The adjustment factors vary from 1.00 to 2.03. While these 
adjustments are expected to reduce the potential bias in the study estimates, they had only a 
small adverse impact on survey precision.^ 

4. Component Four: Part 2 Survey Nonresponse 

We used a propensity modeling approach for the part 2 nonresponse adjustments to take 
maximum advantage of the best available data from the part 1 interview. The propensity score 

^We examined the design effects due to adjusting the weights for nonresponse by comparing 
the coefficient of variation in the weights before and after applying the nonresponse adjustments. 
Nonresponse adjustments to the part 1 interview imparted less than a 10 percent loss in relative 
precision, except in the Graduated Verification Comparison districts, where the loss was about 
13 percent. 
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methodology uses a logistic regression model to predict part 2 response status among part 1 



completes based on indicator variables that describe the student’s characteristics. We then use 
the inverse of the predicted probability of response from the model as the initial adjustment 
(subject to some trimming for outliers) to the weights for the part 2 completes.^ 

As a starting point for developing the modeling procedures as for the part 1 adjustments, we 
reviewed the part 2 response profiles (among part 1 completes) across the various sampling 
frame and part 1 survey outcomes to isolate the factors that appeared to be related to part 2 
nonresponse (Table VII.5). With a propensity-based technique, one needs to consider that the 
relationship between survey response and a student’s characteristic may not be the same across 
categories of another student characteristic. For example, the response rates of free certified 
students in households in which the parent respondent has less than a high school education may 
differ from those of free certified students in households where the parent has some college or a 
college degree. In this circumstance, using one variable for free certified and one variable for 
education level does not adequately compare the variation. To account for such interactions 
under a propensity modeling approach, we can either include interaction terms in the model or 
estimate separate models for the groups. 

Because of the large number of observed characteristics and possible interactions, we 
focused on possible interactions of district membership and meal status with the other 
characteristics in predicting part 2 completion status. We focused on these two characteristics 
because the socioeconomic status of the students and their related meal status are highly 



^We decided against the weighting class approach for the past 2 nonresponse adjustments, 
because the large number of characteristics available would have created more cells than we had 
observations. The propensity model approach uses all the data, and in fitting the model gives the 
various characteristics different weights depending on their association with nonresponse. 
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Below high school 86.4 93.1 81.3 87.5 87.6 

High school graduate 80.8 81.8 86.9 80.8 82.0 

Some college 83.4 80.3 82.8 88.6 83.5 

College graduate 87.9 81.6 87.3 79.6 84.8 
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correlated with the key study outcome. Furthermore, the percentage of free and reduced-price 
students varied across the districts, and districts exhibited different demographic profiles and 
program participation characteristics. The sample sizes by district were small (less than 100 in 
most cases), so to keep the level of review manageable, we collapsed the districts into four 
groups for this review, defined by the combination of the two types of pilot programs (Up-Front 
Documentation or Graduate Verification) and pilot status (pilot or comparison). Our results 
indicated that the response patterns by the four district groups were more similar than the 
response patterns by meal price status categories (see Table VII. 5), with paid status students 
always showing a lower cooperation rate than the free students. Therefore, we decided to 
estimate three models — one for free, one for reduced-price, and one for paid meal price status — 
and to include main effect indicators for each characteristic category. Because the cases 
identified as over 400 percent of FPL in part 1 could not be nonrespondents to part 2, we 
assigned a part 2 nonresponse adjustment factor of one to these cases, and applied the models 
only to the cases under 400 percent of FPL. 

To form the final nonresponse adjustment for the part 2 completes, we began with a full 
variable weighted logistic regression model for each of the three meal price status groups. We 
included in the model estimation all part 1 completes, except those with income exceeding 
400 percent of FPL, assigning a dependent variable that was equal to 1 if the case completed the 
part 2 interview, and 0 otherwise. We estimated the models using the Sudaan software 
procedures, which accounts for the complex sample design in the estimation process. Data were 
weighted by the inverse of the product of the three weighting components developed above. 
From the full models we eliminated nonsignificant factors individually or in groups in succession 
until we achieved a model that contained a set of factors that were significant at about a p-value 
of .30. Because a few of the variables with high, not statistically significant p-values (between 
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.10 and .35) appeared to improve the r-squared values of the model when they were included, we 
retained these variables in the models for the adjustment process. 

We also reviewed the resulting propensity scores for large, potentially influential values. 
We found the maximum values to be 3.02/14.3/1.99 for the free/reduced/paid meal status 
models. Given the large value for the reduced-price meal cases, we explored these adjustments 
further and found that only two scores were above 4 (one at about 6 and another at 14.3). While 
these two values did not appear to adversely affect the variation in the weights, we trimmed them 
to a value 3.90 (the next lower value in the ranking) to prevent any other undo influence on the 
analysis. Overall, the part 2 survey nonresponse adjustments reduced the relative precision in the 
survey estimates by less than 10 percent for all but the Graduated Verification comparison 
districts (in which the relative loss in precision was about 17 percent among the combined meal 
status categories).^ 

Table VII.6 provides a summary of the three models used to create the final part 2 
nonresponse adjustments for the cases under 400 percent of FPL. The table presents a listing of 
the characteristics considered in the models and indicates for each meal status group which 
factors appeared to be influential on the response patterns along with the model coefficients and 
their associated significance levels. The table also presents the Lemeshow goodness-of-fit 
statistic, the significance level, and the r-squared value for each final model, as well as, for 
comparison, the r-squared for the full model prior to variable reduction. 



o 

Based on a comparison of the CV of the weights before and after adjustment. 
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PART 2 NONRESPONSE ADJUSTMENT MODELS 
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Characteristic Variable 

Evaluated in the Models Coefficient P-value Coefficient P- value Coefficient P-value 
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5. Final Adjustments 

After calculating the nonresponse adjustments for part 1 and part 2, we prepared a ratio 
adjustment to align the sum of the final weights for the completed part 1 and part 2 interviews to 
match the sum of the weights obtained for larger full released sample using the product of the 
first and second weight components (the inverse probability of selection and the eligibility rate 
adjustment). With the part 2 propensity modeling approach that adjusts the weights on an 
individual-student basis based on their estimated probability of response, the sum of adjusted 
weights does not always reproduce the original estimates of the population sizes from the 
released sample. Therefore, we used this process to adjust the weights for the completed 
interviews to ensure that they summed to our best estimate of the eligible population. For the 
part 1 completes, since the weighting class approach was designed to reproduce the population 
estimates, this step served merely as a check on the implementation; the adjustment value was 
found to be 1 for all the part 1 completes. 

As a final step in preparing the weights, we applied a ratio adjustment to the completed part 
1 and part 2 interviews in the graduated verification comparison district 3 so that the cases 
reporting to be of Hispanic origin would match in percentage in the comparison district of GV 
pilot district 3. Based on the weights developed above, we found that the pilot district showed 
about 37/34 percent (unweighted/weighted) of the part 1 completes to be of Hispanic origin, with 
the comparison district at 22/17 percent. Therefore, to better align the two districts for 
comparative purposes, we adjusted the part 1 and part 2 weights by a ratio adjustment equal to 
weighted proportion of students in GV pilot district 3 by meal status and four racial categories 
(White non-Hispanic, Black non-Hispanic, Hispanic, and Other, as reported in the part 1 
interview) divided by the equivalent proportion in GV comparison district 3. 
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The final weights for the part 1 eompletes eonsisted of the produet of the first three 
eomponents — the inverse probability of selection, the eligibility rate adjustment, the part 1 
nonresponse adjustment — and the final ratio-stratification adjustment. Likewise, the final 
weights for the part 2 completes consisted of the product of all five components. For GV 
comparison district 3, as indicated above, the weights also received an adjustment to align the 
weighted sample distribution to match that in GV pilot district 3 by meal status and racial profile. 
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VIII. IMPUTATION OF MISSING INCOME DATA 



A primary goal of the survey for the Evaluation of the NSLP Application Verification Pilot 
Projects was to develop an accurate, independent estimate of the income eligibility status of each 
student in the sample. It was critical to classify each household as having income (1) less than 
130 percent of the federal poverty level (FPL) (eligible for free NSLP meals), (2) between 130 
and 185 percent of FPL (eligible for reduced-price NSLP meals), or (3) greater than 185 percent 
of FPL (not eligible for free or reduced-price meals). 

To minimize respondent burden and maximize household participation in the survey, we 
conducted the study survey in two parts. In the part I interview, which was conducted by 
telephone with most respondents and in person with some, respondents reported on their 
experiences with the school lunch program, provided data on the characteristics of their family, 
and estimated their overall income and the total number of persons in their household. From this 
information, we established whether reported household income was above 400 percent of FPL, 
because, if it was, we could confidently classify them as having income above 185 percent of 
FPL (which would make them ineligible for free or reduced-price meals). Households with 
income above 400 percent of FPL were therefore not interviewed further. Households with 
income below 400 percent of FPL were asked to complete the part 2 interview, which entailed 
detailed questions about household composition, economic relationships among persons in the 
household, and information on income by source for each person 16 or older in the household. 

This data collection structure resulted in two different situations in which respondents 
provided partial data on income. First, some respondents completed both the part 1 and part 2 
interview but failed to provide data on one or more items on the part 2 interview necessary to 
calculate total household income. Due to item nonresponse, these cases have partially missing 
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part 2 income. Second, some respondents provided information only for the part 1 interview 
although it was determined in that interview that their income was less than 400 percent of FPL, 
so a complete set of data for the household should include a part 2 interview. Because of unit 
nonresponse to the Part 2 interview, these cases have totally missing part 2 income. 

Of our total sample of 3,685 students, 3,020 students’ families completed the part 1 
interview. Of these 3,020, 401 had income above 400 percent of FPL, so a part 2 interview was 
not attempted; 2,125 completed the part 2 interview; and 494 did not complete the part 2 
interview, although one was attempted. 

In this situation, the analyst can follow one of two courses: (1) use only the completed 
cases, or (2) use the cases with totally missing part 2 data and the cases with partially missing 
part 2 data by imputing the missing data item relevant to calculating total household income. 
Following the first course assures that the data for each case included in the analysis are as 
accurate as possible. However, the households who completed the part 1 interview but failed to 
complete the part 2 interview and households who had item nonresponse on the part 2 interview 
may differ systematically from the households who provided complete part 2 data. Therefore, 
omitting these households with totally or partially missing income data from the analysis creates 
a significant risk of causing bias in the estimates of study outcomes. 

To avoid this risk and to take full advantage of the data collected, we imputed income for 
both the group with partially missing part 2 income (item nonresponse) and the group with 
totally missing part 2 income (unit nonresponse). Section A below describes the imputation 
process for the cases with partially missing part 2 data. Section B describes the imputation 
process for the cases with totally missing part 2 data (but complete part 1 data). In Section C, we 
present the results of the sensitivity analysis we conducted to determine whether our imputation 
methodology affected our key model estimates. 
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A. IMPUTING MISSING INCOME AMOUNTS TO ADDRESS ITEM NONRESPONSE 



I. Objectives and Considerations 

When completing the part 2 interview, the 2,125 parents/guardians were asked a number of 
questions, including some on income for each household member. In particular, they were asked 
to list all persons in the household, and then state, for each person 16 or older, whether that 
person had any income from each of 21 sources during the prior calendar month. For each 
source that a person received, respondents were asked whether documentation was available; if it 
was, they were asked to retrieve it. Finally, they were asked the amount of income for each 
source received. This created a data analysis structure in which the respondent could fail to 
provide data for up to 42 variables (21 source indicators and values for each source for which 
income was non-zero for the previous month) for each adult in the household. In addition, each 
respondent was asked to indicate whether he or she received food stamps or cash assistance for 
low-income fa mi lies with children (TANF). We assumed that if a parent received food stamps 
or TANF benefits, their child was eligible for free meals, so we did not use the income data in 
our analysis, even if the household provided detailed income data.* 

In the 2,125 student households who completed part 2, we identified 4,065 adults, among 
whom current receipt of food stamps/TANF for 774 was reported or documented. Subtracting 
these 774 adults from the subsample of 4,065 left 3,291 adults for whom a response to the 42 
income source and value questions were applicable to our analysis objectives. For 245 sample 
members, one or more adults was missing one or more income items, and we replaced these 
missing item responses with imputed values, while leaving unchanged all complete, consistent 
answers provided for these sample members. We considered responses to the source and value 

*The rationale for this assumption is that food stamp/TANF receipt connotes automatic 
eligibility for free meals regardless of household income. 
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questions to be consistent if income from the source was reported and the amount was non-zero 
or missing. We deemed a set of source and value responses inconsistent if a source was reported 
but the value was zero, and we edited these responses to indicate they did not have income from 
this source. 

We applied one of three imputation methods to responses for sample members if the source 
indicator or the income value was missing: (1) data edits (discussed above), (2) cell-based 
Bernoulli indicator of income source with median value replacement, and (3) sequential hot-deck 

'•y 

imputations for both source and value missing items. Table VIII. 1 presents the item-missing 
counts for each of the 21 income items and shows the methods used to replace the missing 
values. These specific imputation methods are discussed in the following sections. 

Our rationale for selecting either the Bernoulli/median replacement approach or the 
sequential hot-deck imputation approach was based on the level of missing data observed and the 
potential impact the method would have on the analytical results. In general, the Bernoulli/ 
median imputation approach was simple to conduct compared to the hot-deck approach. 
However, the hot-deck method produces individual variation in imputed value responses that 
mimic the variation observed in the reported data."^ In contrast, median value replacement 
generates the same imputed value for all people in the same imputation group. As a result, in 
this study in which the income is used to determine eligibility for free or reduced-price meals, if 
the median value was above/below the threshold, all the missing cases would be assigned under 

^Chapman(1976), Cox (1980), and Kalton and Kaspryzk (1986) discuss these methods. 

We decided for simplicity that if we wanted to use a hot-deck approach for the value 
imputations for an item, we would use a hot-deck approach for the source imputations for that 
item as well. Likewise, if we decided to use a median replacement approach for the value 
imputations for an item, we also used a simpler Bernoulli source imputation process on that item. 

“^The Bernoulli and the hot-deck methods both produce individual variation in the imputed 
values for the source imputations. 
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INCOME ITEM NONRESPONSE AND IMPUTATION METHODS APPLIED 
(BASED ON 3,291 APPLICABLE ADULT RESPONSES) 
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the median imputation methods as being ineligible/eligible and potentially bias the results; the 
cell-based application process serves to reduce such a problem. While the median imputation 
method suffers from introducing an artificially uniform set of responses, the potential for bias 
depends on the level of missing data. As an example, if the number of missing responses is less 
than 1 percent, then at worst case, 1 percent of the cases are assigned to be ineligible/eligible 
when in fact they have the opposite outcome, which would bias the results to the same degree. 
In this study, if the level of missing data (combined source and value) on an item was less than 
0.5 percent (which equates to 16 or fewer missing responses to an item among the 3,291 
applicable adults), we decided that the potential bias from a median replacement strategy was 
acceptable (since it was less than 0.5 percent). We used the simpler technique. 

We found that the missing rates were actually quite low on most items, with only three of 
the 21 income source and value items showing combined missing rates higher than 0.5 percent. 
For two of these items, income for the primary job (having a missing rate of 6.1 percent: 201 
missing source and values among 3,291 adults) and income from a business (having a missing 
rate of 1.6 percent), we used the more data-processing- intensive hot-deck imputation procedures. 
For the third item, income from a farm, the missing rate was only 0.6 percent, and since less than 
1 percent of the adults reported income from this source, we decided to use the median 
replacement methods. For the remaining 18 items, the missing rates were less than 0.5 percent 
and we used the Bernoulli/median replacement approach. 

2. Bernoulli and Median Replacement Imputations 

For 19 of the 21 income source questions, if a respondent who had income from the source 
did not answer, we used a random Bernoulli cell/class-based imputation method that assigned a 
value of 0 (did not have income from the source) or 1 (had income from the source) to impute 
the missing item source answers. For these 19 items, if the income amount was missing, we 



113 




replaced the missing amount with the median value, which we computed from adults in the same 
imputation class with non-missing responses. 

To improve the accuracy of the imputation process, we first formed cells or classes with the 
goal of subdividing the cases with and without missing values into groups that would tend to 
have similar outcomes. We defined these imputation classes by meal price status. If a meal 
price status group contained enough cases, we subdivided the meal status classes by the adult’s 
relationship to the household respondent (three values: l=respondent and presumably the head 
of the household, 2=a spouse or partner of the respondent, and 3=any other adult). In addition, 
for the farm income medians, we used information on whether the respondent indicated a profit 
or loss (when available) to form the classes to produce separate positive and negative median 
values for these situations. 

For the adults in each imputation class who provided a response, we then computed the 
proportion who received income from the source, and the median amount reported by those who 
received income from it. 

Finally, to impute the missing source items for each imputation class, we used the Bernoulli 
statistical distribution to generate a sequence of outcomes of 1 or 0 (having or not having the 
trait) that on average yields a specific proportion possessing the trait. For example, repeated coin 
tosses are a Bernoulli process in which, on average, half the outcomes will be heads and half 
tails. We used a Bernoulli random number generator in the SAS software package to generate 
the required imputed values. For each adult with a missing value, the software generated a value 
of 0 or 1 that based on our specification of the desired proportion of “1” value outcomes would 
in repeated value generations produce values of 1 in proportion to the number of adults having 
the source in the imputation class as observed from the non-missing responses. 
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3. Sequential Hot-Deck Imputation 

For the job and business income items, we used a hot-deck imputation procedure to impute 
missing responses. For each item to be imputed, a basic random hot-deck procedure selects a 
“donor” respondent at random from cases with non-missing values and substitutes the donor’s 
response for the missing value for the recipient. As with the median approach, we perform these 
imputations for classes that are formed in such a manner that the donors and recipients would be 
expected to have similar outcomes. To reduce the chances that the same donor is used 
repeatedly (thereby assigning the same imputed outcome to multiple donors), the classes are 
chosen so that each class has a minimum ratio of donors to recipients. To increase the similarity 
of the donors and recipients, the sequential hot-deck procedure refines the classing step by using 
a card-like sorting process to place similar donors and recipients within each cell together. Each 
recipient is then assigned the value of the donor that precedes it in the “deck.” The sequential 
card-like deck sort and selection process using data from other survey respondents (referred to as 
a “hot” imputation process as opposed to a “cold” imputation process, which might use data from 
external survey sources) gives the method its “hot-deck” name. 

In establishing the classing and sorting criteria, we considered district membership a key 
classification variable, because we expected cases in the same district to have somewhat similar 
income levels. Based on our experience with these imputations, we wanted at least three donors 
per recipient in a cell to reduce the chance that the same donors would be used repeatedly and 
because the sort criteria could restrict the donor pool further. However, we found the ratio of 
donors to recipients was often small in district-based cells. Therefore, we formed the cells either 
by using district membership by itself (because additional criteria would make the cell sizes too 
small) or by using characteristics identified through regression analysis as good predictors of the 
source or income value. 
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To identify other potential predietors of the souree and ineome values, we pooled data aeross 
districts and estimated logistic and linear regression models for each of the two items. ^ We used 
the available part 1 and 2 household and person characteristics as predictors in these models. We 
used a logistic model to predict whether the adult had the income source and a linear regression 
model to predict the value among those with the source. From these analyses, we identified the 
top two or three predictors and attempted to use them in forming the classing cells. If use of 
these variables to form the classing cells resulted in cells with three or fewer donors, we 
collapsed the cell and/or shifted the predictor from classing item to sorting variable. 

With the classes and sorting criteria identified, we first imputed the missing values for the 
cases having non-missing source responses and then jointly imputed the source and value for the 
cases with both missing source and value responses. Under the joint process, the donor selected 
provides both the source indicator and the income value to avoid generating inconsistent pairs of 
responses (such as not having the source yet having a positive value). For the income from the 
primary job, we used as classing variables district membership and relationship of the adult to 
the household respondent, and sorted on whether the adult worked 40 or more hours a week (full 
time vs. part-time) for both the value and joint source and value imputations. For the business 
income questions, for the value-only imputations, we formed the classes based on relationship 
and whether they indicated they had a profit or loss, with a sort on whether the adult had 
completed a college degree. For the source and value imputations, since the relationship of the 



^We prepared these models using all the available characteristics and from the results ranked 
the predictors by their significance level as reflected in the Wald chi-squared test statistic. Since 
these models were prepared to identify the dominate predictors, rather than to form a predictive 
equation, we did not attempt to refine them using stepwise methods or other variable-reduction 
techniques. 
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adult to the respondent dominated the logistic regression model as the predictor of the presence 
of business income, we based the classes solely on this outcome and did not use sorting. 

B. IMPUTATION OF INCOME ELIGIBILITY STATUS FOR CASES TOTALLY 

MISSING PART 2 INCOME 

This section describes the income imputation process for cases for which part 1 income data 
were complete but all part 2 income data were missing. Part 1 respondents who reported 
incomes above 400 percent of FPL (and thus were not contacted to complete a part 2 survey) 
were considered to be ineligible and were assigned to the highest income category (income 
>400) for the analysis. Part 1 respondents who reported receiving food stamps or TANF but did 
not respond to the part 2 survey were assumed to be eligible for free meals and were assigned to 
the lowest income category (income <130) regardless of their reported income amount.^ The 
imputation procedures described below were not performed in these two types of cases. The 
procedures were applied to those households that completed the part 1 survey, reported no food 
stamp/TANF receipt but did report income below 400 percent of FPL, and did not respond to the 
part 2 survey. This approach represents an alternative to the weight based approach for 
addressing unit non-response. 

For these households, we adjusted reported income based on analysis of data from 
households for which we had income based on both the part 1 survey and the part 2 survey, and 
that did not report receiving food stamps or TANF on the part 1 survey. The focus of the 
adjustment procedure was to determine — for households responding to the part 1 survey but not 
the part 2 survey — whether the income category of the household based on part 1 survey data 

^The procedures we outline below provide no basis for imputing the incomes of these 
households, because households that completed both the part 1 and part 2 surveys but also 
reported and documented food stamp or TANF receipt were not asked the full set of income 
questions on the part 2 survey. 
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(less than 130 percent of FPL, 131 to 185 percent of FPL, more than 185 percent of FPL) would 



have been different if we had been able to complete a part 2 survey for the household. This 
adjustment procedure involved six steps: 



1. We determined the subsample with which to examine the relationship between 
income reported on the part 1 survey and income collected during the part 2 survey. 
This sample included all sample members whose households completed both a part 1 
and a part 2 survey, with two exceptions. First, households that provided incomplete 
income information in their part 2 survey (and thus were subject to the imputation 
process described in Section A above) were excluded from this subsample. Second, 
households that reported receiving food stamps or TANF in the part 1 survey were 
also excluded. 

2. For this subsample, we classified their household income reported on the part 2 
survey into one of three categories: (1) income less than 130 percent of FPL and 
thus eligible for free meals, (2) income between 131 and 185 percent of FPL and 
thus eligible for reduced-price meals, and (3) income greater than 185 percent of 
FPL and thus ineligible for free or reduced-price meals. 

3. We estimated two binary logit models using the households within the subsample. 
The dependent variable in the first logit was a binary variable equal to one if the 
household was ineligible for free or reduced-price meals (that is, had income above 
130 percent of FPL) and zero otherwise; the dependent variable in the second logit 
was a binary variable equal to one if the household was ineligible for free meals (that 
is, had income above 185 percent of FPL) and zero otherwise. The estimated 
parameters from each model indicated the effect of the independent variables in the 
model on the probability of the household being in a higher income category 
(ineligible) based on the part 2 survey. The independent variables in the model 
included, most important, a measure of income as reported on the part 1 survey. 
Other independent variables included factors we felt might be related to students’ 
income classification from part 2 after controlling for part 1 income. We describe 
this model in greater detail below. 

4. We selected a sample of households whose income classification was to be imputed. 
This sample included most of the households that responded to the part 1 survey but 
not to the part 2 survey. However, as mentioned above, we did not impute income 
for, or include in the estimation, two groups of households that responded to the part 
1 survey but not the part 2 survey: (1) those with incomes above 400 percent of FPL 
(who would automatically be considered income-ineligible), and (2) those who 
reported receiving food stamps or TANF in the part 1 survey (who would 
automatically be considered income-eligible). 

5. For each household in the group to be imputed, we calculated the following two 
probabilities using the estimated coefficients from the two binary logit models, along 
with the household’s characteristics: (1) PI, the probability that they would have 
been classified as ineligible for free or reduced-price meals (with income exceeding 
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185 percent of FPL) if the part 2 survey had been completed, and (2) P2, the 
probability that they would have been classified as ineligible for free meals (with 
income exceeding 130 percent of FPL) if the part 2 survey had been completed. 
Additional details on the calculation of PI and P2 are provided below. 

6. For each household for which we imputed an income classification, we selected a 
random number between 0 and 1 (from a uniform distribution). If the random 
number was less than PI, the household’s imputed income classification was 
ineligible for free or reduced-price meals. If the random number was greater than PI 
but less than P2, the household’s imputed income classification was eligible for 
reduced-price meals but not free meals. If the random number was greater than P2, 
then the household’s imputed income classification was eligible for free meals. 



1. Estimating the Model 

The key to the imputation procedure was estimating the two logit models, pooled across Up- 
Front Documentation and Graduated Verification sites, and specified as follows: 



IFRR ■ B FRP + UfRP-- 

y u ' y 

where IFRR = 1 if IFRP* > 0 

y 

= 0 otherwise 
where 1F„ =1 if IF* >0 

y 

= 0 otherwise 

In these models, IFi/ is an index variable showing student z’s propensity for being ineligible 
for free meals (income >130% FPL), and IFRPy* is an index variable showing student z’s 
propensity for being ineligible for free or reduced-price meals (income >185% FPL). The 
student was actually ineligible for free meals according to the part two survey if IFij* is greater 
than 0, and this situation is shown by the indicator variable IFij taking on the value 1. 
Analogously, the student was actually ineligible for free or reduced-price meals according to the 
part two survey if IFRPy* is greater than 0, and this situation is shown by the indicator variable 
IFRPij taking on the value 1. The vectors Xpij and XfRPij consist of variables representing 
household characteristics hypothesized to affect student z’s income as reported in the part 2 
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survey. Finally, /3 f and are parameters to be estimated, and and are random-error 
terms that are assumed to have a logistic distribution. 

The independent variables in the models (Xy) are assumed to influence a student’s 
propensity for being classified in higher-income categories (that is, having higher reported part 2 
income). Key variables included in Xy indicate the student’s part 1 income classification relative 
to the FPL — that is, they reflect whether or not a student’s income is close to the relevant free/ 
reduced-price income eligibility thresholds.^ Additional variables included in Xy are: 

• Household size and structure 

• Employment status of household members 

• Free/reduced-price certification status 

• Home and vehicle ownership 

• Race/ethnicity 

• Language spoken at home 

• Parents’ educational attainment 

• Household mobility 

o 

• Pre-pilot NSLP certification status among those in grades 3 and above 

• Parent’s views on the NSLP 

• District indicators 



We calculated each student’s part 1 income relative to the PPL and then created a set of 
income deviation variables. Lor the free/reduced-price eligibility model, these variables included 
a measure of the extent to which income relative to the PPL deviated from the cutoff eligibility 
value for free or reduced-price meals (185 percent of the PPL). We also included a squared 
version of this variable. The free eligibility model included an analogous set of variables, except 
that the 130 percent of the PPL cutoff was used. All other independent variables in the free and 
free/reduced-price models were the same. 

^Por students in kindergarten through 2nd grade, this variable was set to 0 and a separate 
missing value indicator variable was included in the model. 
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2. Imputing Income Categories 

As described above, we estimated f5p and using binary logit models and the sample 
with valid part 1 and part 2 survey information on income. We then took the sample of students 
who (1) completed only a part 1 survey, (2) reported income less than 400 percent of FPL on the 
part 1 survey, and (3) did not report food stamp/TANF receipt in that survey, and imputed their 
income classification as follows: 



1. We calculated Xf-.^f and Xfrp^Sfrp for each student. 



2. We determined 



PI = V\:{lFRP.. = l} = Pr [iFRPj > o}= Pr{-M™,^ < 



XfrPjjPfrp 



l + e 



XFRPjjPPRP 



3. We determined P2 = Pr > o} = Pr ^ 



XFijPF 



\ + e 



XfijPf 



4. We selected a random number Rij from a uniform [0,1] distribution for each student. 

5. If Rij <P1, then we set IFRPij=l. The student was classified as ineligible for free or 
reduced-price meals. Since students ineligible for free/reduced-price meals are also 
ineligible for free meals (PI <P2), then IFij=l as well.^ 

6. Else if PI <Rij <P2, then we set IFRPij=0, IFij=l. The student was classified as 
eligible for free/reduced-price meals but ineligible for free meals (in other words, 
eligible for reduced-price meals). 

7. Else if Rij >P2, then we set IERPij=0, IEij=0. The student was classified as eligible 
for free/reduced-price meals and eligible for free meals. 



Suppose, for example, that the household income reported for a given student was 
120 percent of EPE. When this characteristic, along with the student’s other characteristics, was 



^Since we estimated the two logit models independently, there were a few cases in which we 
found that PI was greater than P2. In each of these cases, however, the values of PI and P2 were 
extremely close to one another (that is, PI was greater than P2 by only a few hundredths or 
thousandths of a percentage point). In these cases, set re-set P2 to the value of PI. 
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used in conjunction with the estimated parameters from the logit models, we might estimate the 
following probabilities for that student: 

PI = V\:{lFRP^. = l}= 0.05 

P2= Pr{/i^. =l}=0.25 

We would impute that student’s income classification for the main impact model as 
ineligible for free/reduced-price meals if the random number we selected was less than 0.05, as 
ineligible for free meals but eligible for reduced-price meals if the random number was between 
0.05 and 0.25, and as eligible for free meals if the random number was greater than 0.25. 

3. Model Performance 

In carrying out these imputation procedures, we estimated both the free/reduced-price 
ineligibility logit model and the free ineligibility logit model using the 1,487 sample members 
who completed both the part 1 and the part 2 surveys and who met the other conditions described 
above. The coefficient estimates from these models are presented in Table VIII.2. The variables 
representing part 1 income were very strong predictors of students’ part 2 income eligibility 
status, but several other variables were also strong predictors even after controlling for part 1 
income. For example, students’ certification status, household size, the employment status of 
household members, and parents’ educational attainment were all statistically significant at the 
1 percent level. We imputed part 2 income status using the methods described in this section for 
450 students — 112 of these students (25 percent) were classified as free eligible, 100 (22 
percent) were classified as reduced-price eligible, and the remaining 238 (53 percent) were 
classified as free/reduced-price ineligible. 

In our sensitivity analysis, we also estimated models that dealt with missing part II income 
information in alternative ways. 
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TABLE VIII.2 



COEFFICIENT ESTIMATES FROM THE PART 2 INCOME IMPUTATION MODELS 

(Standard Errors in Parentheses)^ 





PI: Income >= 185 Percent 


P2: Income >= 130 Percent 




of FPL 


of FPL 


Intercept 


0.768 


0.460 




(0.780) 


(0.832) 



District Pair'’ 


UFD District Pair 1 


— 


— 


UFD District Pair 2 


-0.241 


0.781* 




(0.423) 


(0.466) 


UFD District Pair 3 


-0.437 


0.453 




(0.435) 


(0.463) 


UFD District Pair 4 


0.165 


1.203** 




(0.506) 


(0.501) 


UFD District Pair 5 


-0.616 


0.541 




(0.435) 


(0.455) 


UFD District Pair 6 


-0.309 


0.516 




(0.431) 


(0.466) 


UFD District Pair 7 


-0.312 


0.816* 




(0.432) 


(0.427) 


UFD District Pair 8 


-0.453 


0.611 




(0.417) 


(0.440) 


UFD District Pair 9 


-0.164 


0.560 




(0.452) 


(0.459) 


GV District Pair 1 


0.457 


0.753 




(0.414) 


(0.439) 


GV District Pair 2 


0.055 


0.781* 




(0.432) 


(0.462)* 


GV District Pair 3 


0.369 


1.633*** 




(0.446) 


(0.480) 


Pilot District Status 


Up-Front Documentation Pilot District 


0.538** 


0.133 




(0.213) 


(0.210) 


Graduated Verification Pilot District 


0.060 


-0.703** 




(0.266) 


(0.277) 


Certification Status 


Certified Free 


-1.506*** 


-1.709*** 




(0.238) 


(0.255) 


Certified Reduced Price 


-0.467* 


-1.158*** 




(0.245) 


(0.225) 
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Table VIII.2 {continued) 



PI: Income >= 185 Percent P2: Income >= 130 Percent 

of FPL of FPL 



Part 1 Income Relative to Poverty 



Income as % of FPL - 185 


0.050*** 


0.034*** 




(0.009) 


(0.007) 


(Income as % of FPL - 185) Squared 


0.00028*** 


0.00013** 




(0.00009) 


(0.00005) 


(Income as % of FPL - 185) * 


-0.034*** 


-0.011 


(Binary Indicator of Whether Income > 185% of FPL) 


(0.012) 


(0.012) 


{(Income as % of FPL - 185) * (Binary Indicator of 


-0.00033*** 


-0.00021*** 


Whether Income >185% of FPL)} squared 


(0.00009) 


(0.00006) 


Number of HH members 


-0.357** 


-0.710*** 




(0.142) 


(0.145) 


Number of children in HH 


0.168 


0.416*** 




(0.151 


(0.156) 


Two-parent HH 


-0.229 


0.133 




(0.242) 


(0.242) 


Respondent employed 


-0.459* 


-0.302 




(0.249) 


(0.275) 


# of employed HH members 


1.689*** 






(0.211) 


(0.221) 


Own home 


-0.038 


0.051 




(0.188) 


(0.197) 


Number of vehicles owned 


0.045 


-0.077 




(0.090) 


(0.091) 


Certified in the pre-pilot period 


-0.105 


-0.276 




(0.197) 


(0.200) 


English primary language 


0.203 


-0.614 




(0.446) 


(0.501) 


Respondent’s Education 


Lacks HS diploma 


-0.043 


0.039 




(0.261) 


(0.301) 


Some postsecondary 


0.405** 


0.426** 




(0.189) 


(0.189) 


College degree or more 


-0.001 


0.837*** 




(0.308) 


(0.293) 


Race/Ethnicity 


Black 


-0.194 


-0.109 




(0.290) 


(0.316) 


Hispanic 


0.112 


-0.434 




(0.339) 


(0.370) 


Other 


-0.000 


-0.127 




(0.432) 


(0.432) 
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Table VIII.2 {continued) 



PI: Income >= 185 Percent 
of FPL 


P2: Income >= 130 Percent 
of FPL 


School contacted respondent to apply for certification 


-0.144 


-0.028 




(0.192) 


(0.120) 


Believes application process is fair 


0.475 


0.402 




(0.331) 


(0.300) 


Moved within past 2 years 


-0.068 


-0.106 




(0.137) 


(0.165) 


Changed districts within past 2 years 


-0.292 


-0.219 




(0.293) 


(0.326) 


Mean of Dependent Variable 


0.723 


0.459 


Sample Size 


1487 


1487 



FPL = federal poverty level 

Note: The model also includes binary variables indicating that the following variables were originally missing 

and their values have been imputed using the mean value among non-missing observations: pre-pilot 
certification status, belief that application process is fair, income. 

“None of the standard errors presented in the table have been adjusted to account for the complex sampling design of 
the data set, although the estimate of the overall pilot impact presented in Volume I was adjusted to correct for 
design effects. 

’’Common district pair numbers are used in Chapters III- VIII, and Tables XI.1-XI.8. Chapters IX and X, and Tables 
XI.9-XI.20 use different common district pair designations. 

*Coefficient estimate is significantly different from zero at the .10 level, two-tailed test 

** Coefficient estimate is significantly different from zero at the .05 level, two-tailed test. 

*** Coefficient estimate is significantly different from zero at the .01 level, two-tailed test. 
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As a group, the explanatory variables of the model allowed us to predict students’ part 2 
income status more accurately than our predictions would have been if we had used part 1 
income alone. In particular, to test the model, we used it to compute predicted income categories 
for households that responded to both part 1 and part 2 surveys but without using the part 2 
information in the imputation process. We then compared the imputed income categories to the 
actual income information reported in part 2 for these cases. Our predicted values matched the 
reported values in 63.8 percent of cases with both part 1 and part 2 income. This was a higher 
percentage of cases for which the imputation matched actual part 2 income than we would have 
obtained if we had used the part 1 income data alone, so we proceeded with the imputations. 

C. SENSITIVITY ANALYSIS 

Since the imputations procedures described in sections A and B were used to construct such 
an important analysis variable for this study, we conducted a series of sensitivity tests to 
determine the extent to which the specific procedures used influenced key estimates from the 
model. Tables VIII.3 through VIII.6 present the results of these sensitivity tests. Table VTTT.3 
shows the results of the deterrence analysis in Up-Front Documentation districts. Within the first 
section, the first row shows certification rates among students with income more than 185% FPL 
in pilot and comparison districts (CD_2) using the final version of the income variable. The next 
three rows show this same measure using three alternative methods of imputing part 2 income 
for students who failed to complete part 2. The final three rows in the first section show 
alternative methods for imputing the income sources of individual household members for 
households that completed some but not all of part 2. These alternatives include the hot-deck 
method described in Section A, replacing all missing income sources with zero, and replacing all 
missing income sources with the 90th percentile among those with valid amounts for that income 
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’Variables are defined in Table II.2 of Volume I. 




SENSITIVITY OF ESTIMATES OF CERTIFICATION AMONG INELIGIBLE STUDENTS TO ALTERNATIVE ESTIMATES OF INCOME 

GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS 
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’Variables are defined in Table II.2 of Volume I. 




TABLE VIII.5 



SENSITIVITY OF ESTIMATES OF CERTIFICATION AMONG ELIGIBLE STUDENTS TO ALTERNATIVE ESTIMATES OF INCOME 
UP-FRONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS 





Pilot 


Comparison 




Group of Students/ Alternative Estimate 


Districts 


Districts 


Difference 



Percentage Certified for Free or Reduced-Price 
Meals Among Students with Income <185% 

Students with Income Less Than or Equal to 185% FPL and Not Directly Certified FPL and Not Directly Certified (CB_3a)'^ 



Main Estimate^ 


42.4 


50.5 


8.1 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


43.1 


51.9 


8.8 


predicted probability 


43.8 


53.3 


9.5 


Drop cases with missing estimate of detailed income 


45.5 


52.7 


7.2 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing Person 
Source Amounts 








By hotdecking 


40.2 


51.1 


10.9 


By setting missing person sources amounts to zero 


40.5 


51.1 


10.6 


By setting missing person source amounts to the 90th percentile amount 


42.8 


52.2 


9.4 




Percentage Certified for Free Meals Among 




Students with Income <130% FPL and Not 


Students with Income Less Than or Equal to 130% FPL and Not Directly Certified 




Directly Certified (CB_la)'’ 




Main Estimate^ 


47.3 


53.9 


6.6 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


48.3 


56.9 


8.6 


predicted probability 


48.8 


57.8 


9.0 


Drop cases with missing estimate of detailed income 


50.9 


59.2 


8.3 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing Person 
Source Amounts 








By hotdecking 


44.8 


54.9 


10.1 


By setting missing person sources amounts to zero 


45.4 


54.9 


9.5 


By setting missing person source amounts to the 90th percentile amount 


46.9 


58.0 


11.1 




Percentage Certified for Free or Reduced-Price 




Meals Among Students with Income <130% 


Students with Income Less Than or Equal to 130% FPL and Not Directly Certified 


FPL and Not Directly Certified (CB_2a)'^ 


Main Estimate^ 


53.0 


62.4 


9.4 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


54.1 


65.9 


11.8 


predicted probability 


54.0 


66.4 


12.4 


Drop cases with missing estimate of detailed income 


57.3 


67.5 


10.2 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing Person 
Source Amounts 








By hotdecking 


50.0 


63.9 


13.9 


By setting missing person sources amounts to zero 


50.7 


63.9 


13.2 


By setting missing person source amounts to the 90th percentile amount 


52.9 


66.1 


13.2 



FPL = federal poverty level; n.a. = not applicable; TANF = Temporary Assistance for Needy Families. 

^Income status based on detailed reports of income by person and source; income status for cases missing detailed estimates of income imputed 
using model predictions and assigning to income status probabilistically; meal price status based on school records. 



^Variables are defined in Table II.2 of Volume I. 
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TABLE VIII.6 



SENSITIVITY OE ESTIMATES OE CERTIEICATION AMONG ELIGIBLE STUDENTS TO ALTERNATIVE ESTIMATES OE INCOME 
GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS 





Pilot 


Comparison 




Group of Students/ Alternative Estimate 


Districts 


Districts 


Difference 




Percentage Certified for Free or Reduced-Price 
Meals Among Students with Income <185% FPL 


Students with Income Less Than or Equal to 185% FPL and Not Directly Certified 


and Not Directly Certified (CB_3a)*’ 


Main Estimate^ 


60.4 


72.2 


-11.8 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


61.6 


72.2 


-10.6 


predicted probability 


63.3 


72.1 


-8.8 


Drop cases with missing estimate of detailed income 


64.6 


74.0 


-9.4 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing 
Person Source Amounts 








By hotdecking 


63.5 


71.5 


-8.0 


By setting missing person sources amounts to zero 


63.5 


71.9 


-8.4 


By setting missing person source amounts to the 90th percentile amount 


65.1 


71.6 


-6.5 




Percentage Certified for Free Meals Among Students 
with Income <130% FPL and Not Directly Certified 


Students with Income Less or Equal to Than 130% FPL and Not Directly Certified 




(CB_la)” 




Main Estimate^ 


55.8 


69.2 


-13.4 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


56.4 


69.3 


-12.9 


predicted probability 


56.7 


69.3 


-12.6 


Drop cases with missing estimate of detailed income 


59.2 


70.8 


-11.6 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing 
Person Source Amounts 








By hotdecking 


56.5 


69.9 


-13.4 


By setting missing person sources amounts to zero 


56.2 


70.4 


-14.2 


By setting missing person source amounts to the 90th percentile amount 


61.0 


70.4 


-9.4 




Percentage Certified for Free or Reduced-Price 
Meals Among Students with Income <130% FPL 


Students with Income Less Than or Equal to 130% FPL and Not Directly Certified 


and Not Directly Certified (CB_2a)*’ 


Main Estimate^ 


62.2 


81.1 


-18.9 


Income Status for Cases with Detailed Estimate of Income Missing 








Imputed using simple estimate of total household 

Imputed using model predictions, assigning to income category according to maximum 


62.9 


81.4 


-18.5 


predicted probability 


63.1 


81.4 


-18.3 


Drop cases with missing estimate of detailed income 


64.1 


83.8 


-19.7 


Include Only Cases with Detailed Estimate of Income Available, and Impute Missing 
Person Source Amounts 








By hotdecking 


62.8 


82.5 


-19.7 


By setting missing person sources amounts to zero 


62.6 


83.2 


-20.6 


By setting missing person source amounts to the 90th percentile amount 


65.8 


83.3 


-17.5 



FPL = federal poverty level; n.a. = not applicable; TANF = Temporary Assistance for Needy Families. 

“Income status based on detailed reports of income by person and source; income status for cases missing detailed estimates of income imputed 
using model predictions and assigning to income status probabilistically; meal price status based on school records. 

'’Variables are defined in Table II.2 of Volume I. 
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source. In each of these alternatives, we have dropped cases that failed to respond to part 2 
entirely, and then reweighted the remaining observations so that they are still representative of 
the full population of ineligible households. Table VIII.4 presents these same findings for 
Graduated Verification districts. The second section of Table VIII.3 shows corresponding 
measures for the variable CD_1 (percentage approved for free meals among students with 
income less than 130% FPL). Tables VIII.5 and VIII.6 present analogous sensitivity tests 
focusing on the estimated certification rates among eligible students. 

The general result of the sensitivity tests was that the findings presented in Volume I would 
not have been qualitatively different if we had used different imputation procedures. In 
particular, although the method of imputation affects the estimated certification rate levels 
among ineligible or eligible students, the estimated dijferences between pilot and comparison 
districts in these estimated certification rates are similar, regardless of the imputation method 
employed. 
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IX. IMPACT ANALYSIS ESTIMATION METHODS 



This chapter describes the methods used to determine the impacts of Up-Front 
Documentation and Graduated Verification on deterrence, barriers, accuracy, and targeting 
efficiency. Section A presents the econometric models used to estimate the impacts of the pilots 
on deterrence and barriers. Section B describes how the estimates from the deterrence/barriers 
models were used to obtain estimates of the impacts of the pilots on accuracy and targeting 
efficiency. 

A. ESTIMATING IMPACTS ON DETERRENCE AND BARRIERS 

The logic of the comparison group design calls for estimating the effects of Up-Front 
Documentation and Graduated Verification by comparing key outcomes in the pilot and 
comparison districts. At its most basic level, this could be accomplished by presenting the 
simple mean values of these outcomes in both types of districts. However, outcome differences 
between the two types of districts may have arisen either from the effects of the pilots or from 
other differences between pilot and comparison districts, such as differences in the 
characteristics of the students they served. The process for selecting comparison districts 
described in Chapter II of this volume was designed to minimize differences between the pilot 
and comparison districts. However, it is unlikely that this process resulted in identical pilot and 
comparison districts. Thus, we used a regression model to further account for differences 
between the characteristics of students enrolled in the two types of districts. After these other 
factors were accounted for, any remaining differences between pilot and comparison districts in 
outcomes was attributed to the effect of the pilot. 
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1. General Model 



To estimate the impact of the pilot on both deterrence and barriers, the same general form of 
the statistical model was used. In each case, the primary outcome measure was an indicator of 
whether or not a student was certified for free or free/reduced-price meals, and the independent 
variables were factors hypothesized to influence certification status, including whether the 
student attended a pilot district. The deterrence and barriers models differed in that the sample 
for the deterrence models included only those students ineligible for free or free/reduced-price 
meal benefits, while the barrier models included only groups of students eligible for benefits. 

The general model used to estimate impacts on deterrence/barriers was: 

(1) y,=c + X,b + '^dpP,+^a,{DP,, >-p,\+e, 

j=2 k=l 

where: y, = outcome of interest for student i (free or free/reduced-price certification) 

Xi = vector of characteristics of student i hypothesized to affect outcome y, 

DPij = binary indicator of whether student i attended to the jth pilot-comparison 
district pair 

Pi = binary indicator of whether student i attended a pilot district 
6i = random error term 

In this model, the outcome of interest (measuring the student’s certification status) is 
regressed on a set of student-level characteristics, a set of binary variables (called district pair 
variables) that each represented a pilot district along with its matched comparison district, and a 
set of variables formed by interacting the district pair variables with a binary variable indicating 
whether the student’s district was a pilot district. The coefficients to be estimated in the model 
included a constant term c, a vector b, d 2 through dK , and a i through uk (where K represents the 
number of pilot districts that implemented Up-Front Documentation or Graduated Verification), 
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while 6i is a random-error term representing unobserved faetors that influeneed the outeome of 
interest. This model was estimated using sample weights, to account for the oversampling or 
undersampling of students depending on their preliminary certification status along with 
differential response rates by different groups of students.' 

We estimated separate models to determine the impacts of Up-Front Documentation and 
Graduated Verification. Since there are nine Up-Front Documentation pilot districts (K=9), that 
model included eight binary district pair variables and nine district pair-pilot status interactions. 
There are three Graduated Verification pilot districts (K=3), so that model included two district 
pair variables and three district pair-pilot status interactions. 

The general model shown above allowed for differential effects of each of the pilot 
interventions in each of the districts in which it was implemented. For Up-Front Documentation, 
for example, the model produced nine different estimates of the impact of the pilot on a given 
outcome (the coefficients on the district pair-pilot status interactions, or ai through uk), each 
representing the estimated impact of the pilot in one of the districts in which it was implemented. 
If a I were estimated to be -0.05, for example, this would imply that the intervention in pilot 
district 1 led to a decrease of 5 percentage points in the outcome measure (the probability of 
certification). To estimate the overall impact of Up-Front Documentation (or Graduated 
Verification), we calculated the simple average of all of the pilot district impact estimates. This 
manner of estimating the overall impact gave equal weight to the effect of the pilot intervention 



*See Chapter VII of this volume for a discussion of the sample weights. 
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in each site, regardless of the size of the district or the number of students included in the sample 

'•y 

from the district. 

In the deterrence and barriers models, the dependent variable was binary variable 
representing a student’s certification status. For the estimates presented in Volume I of this 
report, we used a linear probability model (Madalla 1983) to estimate the pilot impacts. This 
model produced an unbiased estimate of the impact of the pilot on the binary indicator of 
whether a given student is certified. One drawback of a linear probability model is that for 
individual observations, it allows the predicted probability of an event occurring to be less than 
zero or greater than one (whereas a model such as a logit model does not allow predicted 
probabilities to be negative or greater than one). We relied on the linear probability model 
because the computation of the standard error of the overall impact of the pilot (which is the 
standard error of the mean of multiple coefficients from the model) was computationally 
intractable with a non-linear model. 



2 

In calculating the standard error of the overall impact estimate, we took into account the 
fact that not all sample observations contributed equally to the overall estimate — those 
observations from districts with larger than average samples were given a bit less weight than 
observations from districts with relatively small samples in calculating the overall impact 
estimate. 

To assess whether the choice of a linear probability model affected our impact estimates, 
we estimated a simplified version of the model (with the estimated impact of the pilot assumed to 
be the same in all pilot districts) using both the linear probability and logit estimation techniques. 
The results we obtained using the two sets of estimation methods were similar. In particular, for 
the ten different outcomes we examined, the logit and OLS impact estimates were within 1.5 
percentage points of each other in each case and within 1.0 percentage points of each other in 
seven of ten cases. The level of statistical significance (significant versus not significant at the 
0.10 level) from the OLS and logit models was the same in nine of ten cases. In the tenth case, 
the estimated impact of Up-Front Documentation on the probability that a student applied for 
benefits, the OLS estimate of the impact (2.9 percentage points, with a t-statistic of 1.87) was 
significant and the logit estimate of this impact (2.1 percentage points, with a t-statistic of 1.36) 
was not significant. 



136 




A key feature of the model shown in equation (1) was its ability to control for the 
characteristics of students and their households. By controlling for these characteristics, we 
could determine whether any differences in outcomes between students in pilot versus 
comparison districts were due to the effects of the pilot or due to differences in the characteristics 
of the students attending the two types of districts (or of the students who happened to be 
selected into our samples). For example, Up-Front Documentation pilot and comparison districts 
differed somewhat in the racial/ethnic distribution of their samples (see Volume I, Chapter III). 
If race/ethnicity also affected the likelihood that a student became certified, then controlling for 
students’ race/ethnicity prior to estimating the impact of the pilot on deterrence/barriers was very 
important. The characteristics controlled for in the model included variables representing the 
following: 

• Household size 

• Household structure 

• Household income 

• Public assistance receipt 

• Public housing residence 

• Employment status of household members 

• Home and vehicle ownership 

• Race/ethnicity 

• Primary language spoken at home 

• Grade level 

• Pre-pilot certification status 

• Parent/guardian’s educational attainment 

• Household mobility 

• Parent/guardian’s views on administration of school lunch program 
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Students’ perceptions on taste/amount of school meals 






All of the variables included as control variables in the model were characteristics of 
students or their households rather than characteristics of their districts. It is possible that district 
characteristics, such as their experience using direct certification to certify children for free 
meals or the extent to which districts promoted certification among their students, could 
influence the likelihood that students in a given district were certified for free or reduced-price 
meals. However, since these characteristics had the same value for all students enrolled in a 
particular district, and since we estimated a separate impact in each Up-Front Documentation or 
Graduated Verification pilot district, there were no remaining degrees of freedom in the model 
and district characteristics could not be included as control variables. 

The lack of district characteristics in the model is offset by two factors. First, our selection 
of comparison districts (described in Chapter II of this volume) was designed so that these 
comparison districts had characteristics as similar as possible to those of pilot districts. Pilot and 
comparison districts were matched on the basis of both observable characteristics measured by 
publicly available data (such as certification rates, participation rates, and race/ethnicity) and by 
less tangible characteristics provided to us by local observers. Second, some district 
characteristics can be proxied for by student characteristics. In particular, by controlling for 
students’ pre-pilot certification status, we captured differences between pilot and comparison 
districts in their propensity to promote certification among their students."^ 

"^If we had specified the model such that the impact of the pilot was assumed to be the same 
across all district pairs, we could have included a small number of district characteristics as 
control variables in theory. We explored the sensitivity of our results to a model such as this in 
which we controlled for: (1) the pre-pilot certification rate of the district; (2) the pre-pilot 
poverty rate among students in the district; and (3) the degree of centralization of NSLP 
administration in the district. In practice, we found that the estimation results became extremely 
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2. Deterrence 



Two separate versions of equation (1) were estimated to examine the impact of Up-Front 
Documentation on deterrence, and an analogous set of models were estimated to examine the 
impact of Graduated Verification. These models differed according to the sample on which the 
model was estimated. The deterrence models were: 



1. Certification Among Students Not Eligible for Free or Reduced-Price Meals 
(CD_2). The sample for this model included only students ineligible for free or 
reduced-price meals; that is, those with household incomes exceeding 185 percent of 
the federal poverty level. Students who had been directly certified were excluded. 

The model’s dependent variable was a binary indicator of whether the student was 
certified for free or reduced-price meals. This is measure CD_2 shown in Tables 
IV. 1, IV.2, IV.9, and IV. 10 of Volume I. 

2. Certification Among Students Not Eligible for Free Meals (CD_1). The sample for 
this model included students ineligible for free meals; that is, those with household 
incomes exceeding 130 percent of the federal poverty level (and not receiving food 
stamp, TANF, or FDPIR benefits), again excluding directly certified students. The 
model’s dependent variable was a binary indicator of whether the student was 
certified for free meals. This is measure CD_1 shown in Tables IV. 1, IV.2, IV.9, 
and IV. 10 of Volume I. 

The full set of coefficient estimates from the model are shown in Table IX. 1. 

As noted above, the coefficients obtained from the estimation of equation (I) include a 
separate indicator of the impact of the pilot in each of the districts that implemented the pilot. To 



(continued) 

unstable when these district characteristics were added. The estimated effect of the pre-pilot 
certification rate on the likelihood that a student was certified as of our survey varied wildly 
depending on whether we were looking at the Up-Front Documentation districts or Graduated 
Verification districts. The estimate of this effect was also negative, which is counterintuitive. 
We concluded that because the model contained few degrees of freedom (because we had 
relatively few districts in our sample and were already controlling for the district pair variables), 
and because we had already matched pilot and comparison districts well as evidenced by our 
examination of pre-pilot district characteristics in pilot versus comparison districts, there was not 
sufficient variation in these district characteristics to reliably estimate and control for their 
effects. 
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TABLE IX. 1 



COEFFICIENT ESTIMATES FROM THE MODELS OF DETERRENCE 
(Standard Errors in Parentheses)^ 



Up-Front Documentation Model Graduated Verification Model 



Free/RP Meal 
Certification 
Among 
Ineligible 
Students (CD_2) 



Free Meal 
Certification 
Among 
Ineligible 
Students (CD_1) 



Free/RP Meal 
Certification 
Among 
Ineligible 
Students (CD_2) 



Free Meal 
Certification 
Among 
Ineligible 
Students (CD_1) 



Intercept 

District Pair'’ 

UFD District Pair 1 

UFD District Pair 2 

UFD District Pair 3 

UFD District Pair 4 

UFD District Pair 5 

UFD District Pair 6 

UFD District Pair 7 

UFD District Pair 8 
UFD District Pair 9 

GV District Pair 1 
GV District Pair 2 

GV District Pair 3 

District Pair-Pilot Interaction 
UFD District Pair 1 

UFD District Pair 2 

UFD District Pair 3 

UFD District Pair 4 

UFD District Pair 5 

UFD District Pair 6 

UFD District Pair 7 

UFD District Pair 8 



0.159* 


0.172** 


(0.071) 


(0.060) 


-0.042 


-0.030 


(0.034) 


(0.031) 


-0.056 


-0.021 


(0.034) 


(0.029) 


-0.042 


-0.020 


(0.035) 


(0.030) 


-0.068* 


-0.053* 


(0.033) 


(0.028) 


-0.059* 


-0.039 


(0.033) 


(0.030) 


-0.071* 


-0.022 


(0.033) 


(0.030) 


-0.053 


-0.025 


(0.035) 


(0.032) 


-0.026 


-0.030 


(0.034) 


(0.030) 


-0.001 


-0.009 


(0.036) 


(0.031) 


0.047 


0.052* 


(0.034) 


(0.028) 


0.027 


-0.002 


(0.034) 


(0.029) 


0.015 


0.018 


(0.032) 


(0.028) 


-0.012 


-0.015 


(0.032) 


(0.029) 


0.039 


-0.011 


(0.033) 


(0.029) 


-0.032 


-0.032 


(0.033) 


(0.030) 


-0.050 


-0.052 


(0.034) 


(0.030) 



0.488** 0.437** 

(0.153) (0.118) 



-0.005 


-0.059 


(0.049) 


(0.042) 


-0.044 


-0.022 


(0.049) 


(0.042) 
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Table IKA(continued) 



Up-Front Documentation Model Graduated Verification Model 





Free/RP Meal 


Free Meal 


Free/RP Meal 


Free Meal 




Certification 


Certification 


Certification 


Certification 




Among 


Among 


Among 


Among 




Ineligible 


Ineligible 


Ineligible 


Ineligible 




Students (CD_2) 


Students (CD_1) 


Students (CD_2) 


Students (CD_1) 


UFD District Pair 9 


-0.036 


-0.022 








(0.037) 


(0.032) 






GV District Pair 1 


— 


— 


-0.044 


-0.015 








(0.042) 


(0.036) 


GV District Pair 2 


— 


— 


-0.026 


0.002 








(0.044) 


(0.038) 


GV District Pair 3 


— 


— 


-0.004 


0.013 








(0.038) 


(0.033) 


Number of HH members 


-0.009 


0.002 


0.003 


-0.024 




(0.011) 


(0.009) 


(0.026) 


(0.021) 


Number of children in HH 


0.027* 


0.015 


0.016 


0.028 




(0.012) 


(0.010) 


(0.028) 


(0.022) 


Two-parent HH 


0.013 


0.015 


-0.065 


0.009 




(0.019) 


(0.016) 


(0.038) 


(0.031) 


Income 










150-185% FPL 


— 


-0.135** 


— 


0.196** 






(0.026) 




(0.047) 


185-200% FPL 


— 


-0.128** 


— 


-0.282** 






(0.036) 




(0.059) 


200-250% FPL 


-0.035 


-0.144** 


-0.298** 


-0.336** 




(0.034) 


(0.026) 


(0.057) 


(0.048) 


250-300% FPL 


-0.118*** 


-0.178** 


-0.352** 


-0.339** 




(0.034) 


(0.026) 


(0.056) 


(0.048) 


300-350% FPL 


-0.131** 


-0.183** 


-0.336** 


-0.328** 




(0.036) 


(0.029) 


(0.063) 


(0.055) 


350-400% FPL 


-0.090* 


-0.164** 


-0.360** 


-0.336** 




(0.038) 


(0.031) 


(0.061) 


(0.054) 


>400% FPL 


-0.120** 


-0.170** 


-0.362** 


-0.340** 




(0.035) 


(0.027) 


(0.056) 


(0.049) 


Respondent employed 


-0.008 


0.005 


-0.004 


-0.048 




(0.019) 


(0.016) 


(0.047) 


(0.037) 


# of employed HH members 


0.012 


0.004 


-0.020 


0.019 




(0.013) 


(0.012) 


(0.028) 


(0.023) 


Own home 


-0.022 


-0.034* 


-0.017 


-0.024 




(0.018) 


(0.015) 


(0.035) 


(0.028) 


Number of vehicles owned 


-0.008 


-0.005 


0.013 


-0.003 




(0.005) 


(0.004) 


(0.011) 


(0.010) 


Grade of Student 










Kinderg arten/Pre -K 


-0.050 


-0.053 


-0.017 


-0.028 


U‘ to 2"'' 


(0.037) 


(0.031) 


(0.077) 


(0.062 


-0.051 


-0.021 


-0.071 


-0.097 


3’'^ to 5* 


(0.031) 


(0.026) 


(0.069) 


(0.057) 


-0.026 


-0.024 


0.040 


0.011 


00 

O 


(0.018) 


(0.015) 


(0.035) 


(0.030 


-0.010 


-0.021 


0.018 


0.028 




(0.017) 


(0.015) 


(0.032) 


(0.028) 
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Table IKA(continued) 





Up-Eront Documentation Model 


Graduated Verification Model 




Eree/RP Meal 
Certification 
Among 
Ineligible 
Students (CD_2) 


Eree Meal 
Certification 
Among 
Ineligible 
Students (CD_1) 


Eree/RP Meal 
Certification 
Among 
Ineligible 
Students (CD_2) 


Eree Meal 
Certification 
Among 
Ineligible 
Students (CD_1) 


Certified in the pre-pilot period 


0.228** 


0.132** 


0.120** 


0.066* 




(0.022) 


(0.017) 


(0.036) 


(0.029) 


English primary language 


-0.041 


-0.016 


-0.149 


-0.110 


Respondent’s Education 


(0.036) 


(0.033) 


(0.105) 


(0.071) 


Lacks HS diploma 


0.001 


0.003 


0.042 


0.030 




(0.027) 


(0.022) 


(0.065) 


(0.047) 


Some postsecondary 


0.006 


0.009 


0.005 


-0.051* 




(0.014) 


(0.012) 


(0.028) 


(0.024) 


College degree or more 


0.005 


0.006 


-0.028 


-0.062* 




(0.016) 


(0.014) 


(0.042) 


(0.030) 


Race/Ethnicity 


Black 


0.012 


0.005 


0.024 


0.035 




(0.026) 


(0.023) 


(0.042) 


(0.035) 


Hispanic 


0.012 


0.013 


0.092 


0.025 




(0.031) 


(0.026) 


(0.056) 


(0.043) 


Other 


0.002 


-0.002 


0.009 


0.071 




(0.028) 


(0.025) 


(0.069) 


(0.059) 


School contacted respondent to 


-0.015 


-0.022 


0.021 


0.040 


apply for certification 


(0.14) 


(0.012) 


(0.026) 


(0.022) 


Level of satisfaction with 


0.009 


0.004 


-0.013 


0.032 


school lunch service 


(0.022) 


(0.020) 


(0.053) 


(0.042) 



Believes application process is 


0.047* 


0.035 


0.079* 


0.088* 


fair 


(0.023) 


(0.019) 


(0.039) 


(0.035) 


Moved within past 2 years 


0.026 


0.026* 


0.055* 


0.051* 




(0.014) 


(0.012) 


(0.025) 


(0.021) 


Changed districts within past 2 


-0.041 


-0.024 


-0.081 


-0.053 


years 


(0.028) 


(0.024) 


(0.064) 


(0.052) 


R^ 


0.198 


0.166 


0.291 


0.263 


Mean of Dependent Variable 


0.039 


0.038 


0.073 


0.080 


Sample Size 


1,057 


1,377 


423 


629 



FPL = federal poverty level 

Note; Directly certified students were excluded from the model. The model also includes binary variables 
indicating that the following variables were originally missing and their values have been imputed using the 
mean value among non-missing observations: pre-pilot certification status, satisfaction with school lunch 
service, belief that application process is fair, income. 
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Table IKA(continued) 



“None of the standard errors presented in the table have been adjusted to account for the complex sampling design of 
the data set, although the estimate of the overall pilot impact presented in Volume I was adjusted to correct for 
design effects. 

’’Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 

*Coefficient estimate is significantly different from zero at the .05 level, two-tailed test 
** Coefficient estimate is significantly different from zero at the .01 level, two-tailed test. 
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determine the overall impact, we calculated the simple mean of these site-specific impacts. We 
then used the coefficient estimates from the model to calculate the regression-adjusted mean 
outcome among pilot district students and among comparison district students. To calculate the 
regression-adjusted mean in pilot districts, we used the following procedure: 



1 . For each student in any district in the sample, we assumed that the student was in a 
pilot district by setting the value of P equal to 1 for that student. We did not change 
any of the other characteristics of that student or his/her household. 

2. Again for each student, we multiplied the values of their characteristics (including 
the assumed value of P=l) by the estimated coefficients from the model according to 
the formula provided by equation (1).^ For a given student, the value of this sum of 
student characteristics times coefficients represented the predicted likelihood of that 
student’s being certified. 

3. We calculated the mean (weighted, using sample weights) of this predicted 
likelihood across all students in the sample to determine the regression- adjusted 
probability of being certified/applying for benefits among ineligible students in pilot 
districts. 



The regression- adjusted mean in comparison districts was calculated using an analogous 
methodology, except that the value of P for each was assumed to be 0 instead of 1. The 
difference between the regression-adjusted mean in pilot districts versus comparison districts for 
a particular outcome was equal to the estimated impact, by definition. 

These relationships can be written in terms of conditional probabilities. As an example, the 
estimated impact of the pilot on free meal certification among ineligible students can be written: 



(2) Imp { Deterrencell; >130} = Pr {CF}. =117, > 1 30, 7} = 1} - Pr {CF}. =117; > 1 30, 7} = 0} 
where: 

(3) Pr {CF;. = 1 1 7; > 130, 7} = 1} = c + xfi + a, [DP,, * 1] 

j=2 k=l 



^For the purposes of this calculation, we ignored the random error term, effectively 
assuming that it was equal to 0 for each student. 
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(4) Pr{Ci^. =11/,. >130,/^ = 0} = c + Z,i + J^J,D/^. 

y=2 

In other words, the impaet of the pilot (P) on the rate of free eertifieation (CF) among 
students ineligible for free meals (I>130; or income above 130 percent of the federal poverty 
level) is equal to the conditional probability that a given student is certified for free meals given 
that they are ineligible and in a pilot district minus the conditional probability that they are 
certified free given that they are ineligible and in a comparison district. For individual students, 
these conditional probabilities can be calculated by using the coefficient estimates from equation 
(1) where the dependent variable is a binary indicator of whether the student was certified for 
free meals and the sample is restricted to those ineligible for free meals. ^ 

3. Barriers 

To determine the impacts of the pilots on barriers among non-directly certified students, we 
estimated four models each for Up-Front Documentation and Graduated Verification. These 
models were: 



1. Free Certification Among Students with Income < 130% FPL and Not Directly 
Certified (CB_la). The sample for this model included only students eligible for 
free meals; that is, those with household incomes of no more than 130 percent of the 
federal poverty level. Students who had been directly certified were excluded. The 
model’s dependent variable was a binary indicator of whether the student was 
certified for free meals. This is measure CB_la shown in Tables IV. 3, IV. 4, IV. 11 
and IV. 12 of Volume I. 

2. Free and Reduced-Price Certification Among Students with Income < 130% FPL 
and Not Directly Certified (CB_2a). The sample for this model included only 
students eligible for free meals; that is, those with household incomes of no more 
than 130 percent of the federal poverty level. Students who had been directly 
certified were excluded. The model’s dependent variable was a binary indicator of 
whether the student was certified for free or reduced-price meals. This is measure 
CB_2a shown in Tables IV. 3, IV. 4, IV. 11, and IV. 12 of Volume I. 



^All of the terms in equations 2 through 4 also are conditional on the student not being 
directly certified. 



145 




3. Free and Reduced-Price Certification Among Students with Income < 185% FPL 
and Not Directly Certified (CB_3a). The sample for this model included only 
students eligible for free or reduced-price meals; that is, those with household 
incomes of no more than 185 percent of the federal poverty level. Students who had 
been directly certified were excluded. The model’s dependent variable was a binary 
indicator of whether the student was certified for free or reduced-price meals. This is 
measure CB_3a shown in Tables IV. 3, IV. 4, IV. 1 1, and IV. 12 of Volume I. 

4. Free Certification Among FS/TANF Recipients Who Are Not Directly Certified 
(CB_4a). The sample for this model included only students eligible for free meals 
on the basis of their FS/TANF receipt. Students who had been directly certified were 
excluded. The model’s dependent variable was a binary indicator of whether the 
student was certified for free meals. This is measure CB_4a shown in Tables IV. 3, 
IV.4, IV. 1 1, and IV. 12 of Volume I. 



We used the same procedures to calculate the regression-adjusted pilot and comparison 
district mean outcomes along with the impact as was described above for deterrence. These 
estimated impacts of the pilots on these measures of barriers are presented in Volume I, 
Chapter IV. The full set of coefficient estimates from the model are shown in Table IX. 2. 

In terms of predicted probabilities, the impact of the pilot on barriers (in this case, on free 
meal certification among non-directly certified students eligible for free meals) can be written: 

(5) Imp{BarrierslI, < 130} = ^r{CF. = 1 1 /,. < 130,/} = l}-Pr{CF;. = 1 1 /, < 130,/} = 0} 
where: 

(6) Pr {CF;. = 1 1 /, < 130, /} = 1} = c + Xf + +t,a,[Df, * 1] 

>2 k=l 

(7) Pr{CF;. = 1 1 /, < 130, P.=0} = c + X.b + Y,djDP.. 

j=2 

In this case, the model parameters shown in equations 6 and 7 are obtained from estimating 
a version of equation 1 in which the sample is restricted to those eligible for free meals and the 
dependent variable is a binary variable indicating whether the student is certified for free meals. 
The impact of the pilots on the other measures of barriers can be determined analogously. 
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TABLE IX.2 



COEFFICIENT ESTIMATES FROM THE MODELS OF BARRIERS 
(Standard Errors in Parentheses)^ 



Up-Front Documentation Model 


Graduated Verification Model 


Free/RP Meal 
Certification 
Among Eligible 
Students (CB_3a) 


Free Meal 
Certification 
Among Eligible 
Students (CB_la) 


Free/RP Meal 
Certification 
Among Eligible 
Students (CB_3a) 


Free Meal 
Certification 
Among Eligible 
Students (CB_la) 



Intercept 


0.307 


0.316 


0.076 


0.018 


(0.162) 


(0.207) 


(0.165) 


(0.199) 


Disti'ict Pair'’ 


UFD District Pair 1 


0.107 


0.178 








(0.085) 


(0.109) 






UFD Disti'ict Pair 2 


0.129 


0.135 








(0.082) 


(0.110) 






UFD District Pair 3 


-0.036 


-0.028 








(0.085) 


(0.111) 






UFD District Pair 4 


0.102 


0.155 








(0.081) 


(0.110) 






UFD District Pair 5 


-0.027 


-0.088 








(0.087) 


(0.115) 






UFD District Pair 6 


0.179 * 


0.279 * 








(0.086) 


(0.112) 






UFD District Pair 7 


0.121 


0.202 








(0.104) 


(0.137) 






UFD District Pair 8 
UFD District Pair 9 


0.048 


0.086 








(0.084) 


(0.110) 






GV District Pair 1 
GV District Pair 2 






-0.258 ** 


-0.104 








(0.072) 


(0.094) 


GV District Pair 3 






-0.114 


-0.038 








(0.076) 


(0.098) 


District Pair-Pilot Interaction 










UFD District Pair 1 


-0.282 ** 


-0.267 ** 








(0.080) 


(0.102) 






UFD District Pair 2 


0.027 


0.133 








(0.073) 


(0.101) 






UFD District Pair 3 


0.055 


-0.018 








(0.080) 


(0.107) 






UFD District Pair 4 


-0.131 


-0.145 








(0.085) 


(0.115) 






UFD District Pair 5 


0.168 


0.328 ** 








(0.086) 


(0.117) 






UFD District Pair 6 


-0.146 


-0.211 








(0.086) 


(0.116) 






UFD District Pair 7 


-0.197 


-0.150 








(0.103) 


(0.139) 






UFD Disti'ict Pair 8 


0.083 


-0.034 








(0.090) 


(0.120) 






UFD Disti'ict Pair 9 


-0.229 ** 


-0.193 * 








(0.075) 


(0.093) 
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Table YK.2(continued) 



Up-Front Documentation Model Graduated Verification Model 



Free/RP Meal 


Free Meal 


Eree/RP Meal 


Eree Meal 


Certification 


Certification 


Certification 


Certification 


Among Eligible 


Among Eligible 


Among Eligible 


Among Eligible 


Students (CB_3a) 


Students (CB_la) 


Students (CB_3a) 


Students (CB_la) 



GV District Pair 1 






-0.171 ** 


-0.097 








(0.059) 


(0.073) 


GV District Pair 2 






0.012 


-0.049 








(0.058) 


(0.072) 


GV District Pair 3 






-0.112 


-0.253 * 








(0.074) 


(0.107) 


Number of HH members 


-0.089 ** 


-0.072 ** 


-0.046 


-0.255 




(0.022) 


(0.028) 


(0.028) 


(0.138) 


Number of children in HH 


0.116 ** 


0.082 ** 


0.078 * 


0.126 ** 




(0.024) 


(0.031) 


(0.031) 


(0.038) 


Two-parent HH 


0.062 


0.065 


0.065 


0.097 




(0.036) 


(0.048) 


(0.047) 


(0.059) 


Income 










0-50% PPL 


0.099 


-0.068 


0.087 


0.017 




(0.055) 


(0.059) 


(0.075) 


(0.078) 


50-100% PPL 


0.242 ** 


0.099 * 


0.123 * 


0.052 




(0.040) 


(0.050) 


(0.051) 


(0.064) 


100-130% PPL 


0.231 ** 




0.071 






(0.045) 




(0.063) 




130-150% PPL 


0.203 ** 




0.046 






(0.049) 




(0.063) 




Respondent employed 


-0.014 


0.057 


0.074 


0.092 




(0.039) 


(0.051) 


(0.052) 


(0.066) 


# of employed HH members 


0.002 


0.074 


-0.050 


-0.051 




(0.030) 


(0.040) 


(0.042) 


(0.054) 


Own home 


-0.088 * 


-0.090 


0.029 


0.087 




(0.035) 


(0.047) 


(0.044) 


(0.057) 


Number of vehicles owned 


0.029 * 


0.058 ** 


-0.035 


-0.040 




(0.014) 


(0.019) 


(0.020) 


(0.026) 


Grade of Student 










Kindergarten/Pre-K 


-0.208 ** 


-0.076 


0.130 


0.236 * 


U‘ to 2"'* 


(0.078) 


(0.100) 


(0.093) 


(0.118) 


0.219 ** 


0.326 ** 


-0.003 


0.298 ** 


3''* to 5* 


(0.067) 


(0.091) 


(0.084) 


(0.108) 


0.145 ** 


0.239 ** 


0.210 ** 


0.177 ** 


o 

00 


(0.042) 


(0.056) 


(0.053) 


(0.067) 


0.041 


0.124* 


0.098 


0.046 




(0.042) 


(0.055) 


(0.051) 


(0.065) 


Certified in the pre-pilot 


0.297 ** 


0.245 ** 


0.278 ** 


0.175 ** 


period 


(0.036) 


(0.048) 


(0.047) 


(0.060) 


English primary language 


-0.210 * 


-0.212 


0.152 * 


0.182 * 




(0.102) 


(0.129) 


(0.069) 


(0.084) 


Respondent’s Education 










Lacks HS diploma 


-0.045 


0.034 


0.010 


-0.012 




(0.042) 


(0.051) 


(0.049) 


(0.059) 


Some postsecondary 


0.042 


0.050 


-0.023 


-0.021 




(0.035) 


(0.048) 


(0.042) 


(0.057) 


College degree or more 


-0.149 ** 


-0.113 


0.008 


0.063 




(0.054) 


(0.073) 


(0.076) 


(0.101) 
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Table IK.2(continued) 





Up-Front Documentation Model 


Graduated Verification Model 




Free/RP Meal 
Certification 
Among Eligible 
Students (CB_3a) 


Free Meal 
Certification 
Among Eligible 
Students (CB_la) 


Free/RP Meal 
Certification 
Among Eligible 
Students (CB_3a) 


Free Meal 
Certification 
Among Eligible 
Students (CB_la) 


Race/Ethnicity 


Black 


-0.009 


-0.047 


-0.076 


0.010 




(0.063) 


(0.082) 


(0.058) 


(0.075) 


Hispanic 


0.002 


-0.070 


0.131 * 


0.172* 




(0.066) 


(0.089) 


(0.063) 


(0.082) 


Other 


-0.107 


-0.116 


-0.182 


0.003 




(0.072) 


(0.092) 


(0.096) 


(0.123) 


School contacted respondent 


0.001 


0.008 


-0.030 


-0.094 


to apply for certification 


(0.034) 


(0.046) 


(0.039) 


(0.050) 


Level of satisfaction with 


0.004 


0.150* 


0.010 


0.211 


school lunch service 


(0.055) 


(0.073) 


(0.067) 


(0.089) 


Believes application process 


0.211 ** 


0.159 * 


0.240 ** 


0.120 


is fair 


( 0.053) 


(0.073) 


(0.072) 


(0.095) 


Moved within past 2 years 


0.003 


0.024 


0.034 


0.038 




(0.025) 


(0.030) 


(0.026) 


(0.032) 


Changed districts within 


-0.020 


-0.031 


0.099 


0.079 


past 2 years 


(0.051) 


(0.065) 


(0.060) 


(0.072) 


Received TANF during 


-0.070 


-0.088 


0.050 


0.093 


previous month 


(0.072) 


(0.075) 


(0.064) 


(0.067) 


Received Foodstamps 


0.123 ** 


0.144 ** 


-0.050 


0.022 


previous month 


(0.041) 


(0.045) 


(0.051) 


(0.055) 


Received Public Housing 


0.063 


0.057 


0.130* 


0.148 * 


Assistance past month 


(0.064) 


(0.073) 


(0.057) 


(0.063) 


R^ 


0.421 


0.398 


0.251 


0.245 


Mean of Dependent 


Variable 


0.474 


0.512 


0.672 


0.646 


Sample Size 


881 


561 


642 


436 



FPL = federal poverty level. 

Note: Directly certified students are excluded from the model. The model also includes binary variables indicating that 

the following variables were originally missing and their values have been imputed using the mean value among 
non-missing observations: pre-pilot certification status, satisfaction with school lunch service, belief that 
application process is fair, income. 

“None of the standard errors presented in the table have been adjusted to account for the complex sampling design of the data 
set, although the estimate of the overall pilot impact presented in Volume I was adjusted to correct for design effects. 

'’Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 

* Coefficient estimate is significantly different from zero at the .05 level, two-tailed test 
** Coefficient estimate is significantly different from zero at the .01 level, two-tailed test. 
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4. Including Directly Certified Students 

As noted above, the indicator of barriers we used was the rate of certification among groups 
of students eligible for free or for free or reduced-price meals, excluding directly certified 
students. In Volume I, we presented information on barriers both without controlling for 
household characteristics and with such controls for household characteristics. In other words, 
we presented estimates of the simple mean certification rates among eligible students and the 
regression-adjusted mean certification rates among eligible students. The simple mean estimates 
were based on observed certification rates among our sample of non-directly certified students 
eligible for benefits. Since we assumed that directly certified students are eligible for the 
benefits they are receiving, however, their exclusion affects the estimated certification rates. If 
we assumed that all directly certified students were eligible, for example, the certification rate 
among non-directly certified students would be lower than the certification rate among all 
eligible students. Excluding directly certified students, in other words, could make the extent of 
certification barriers appear larger than it is in reality. Thus, to estimate certification rates among 
eligible students, we used the information available to us on the proportion of students in the 
district who were directly certified and assumed that all directly certified students were eligible 
for free meal benefits. Table IX. 3 presents the estimates by district. 

For the barriers tables in Chapter IV of Volume I, we estimated the probability of being 
certified for free meals among all students with household incomes below 130 percent of poverty 
(measure CB_lb in Table II.2 of Volume I), whether directly certified or not, by using the 
following relationships: 

(8) Pr(cert free I inc<130) = Pr (cert free & no DC I inc<130) -i- Pr(DC I inc<130) 

= Pr (cert free I no DC, inc<130)*Pr(no DC I inc<130) -i- Pr(DC I inc<130) 

= Pr (cert free I no DC, inc<130)*[l-Pr(DC I inc<130)] -i- Pr(DC I inc<130) 

= Pr (cert free I no DC, inc<130)*{ l-[Pr(DC)/Pr(inc<130)] } -i-Pr(DC)/Pr(inc<130) 
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TABLE IX.3 



ESTIMATES OE PERCENTAGE OE STUDENTS DIRECTLY CERTIEIED 







Pilot Districts 




Comparison Districts 


Pilot/Comparison Districts 


Total 

Enrollment 


Students 

Directly 

Certified’’ 


Proportion of 
Students 
Directly 
Certified Free 


Total 

Enrollment' 


Students 
Directly 
“ Certified” 


Proportion of 
Students 
Directly 
Certified Free 


Up-Front Documentation 

Blue Ridge/Montrose (PA) 


1,238 


0 


0.000 


1,925 


74 


0.038 


East Stroudsburg/Easton (PA) 


7,504 


421 


0.056 


7,887 


617 


0.078 


Pleasant Valley/Bangor (PA) 


6,676 


159 


0.024 


3,455 


82 


0.024 


Stroudsburg/Pottsgrove (PA) 


5,336 


0 


0.000 


1,989 


0 


0.000 


Maplewood/Newton Falls (OH) 


1,097 


0 


0.000 


1,600 


0 


0.000 


Salem City/Lisbon (OH) 


2,481 


0 


0.000 


1,191 


0 


0.000 


Creve Coeur/North Pekin (IL) 


750 


68 


0.091 


605 


7 


0.012 


Oak Park and Forest River/ 
Valley View (IL)“’ 


2,962 


33 


0.011 


1,774 


17 


0.010 


Williamson County/Wilson County 
(TN)“ 


6,296 


376 


0.060 


10,837 


304 


0.028 


Mean across districts 






0.027 






0.021 



Graduated Verification 



DGF/Breckenridge/Lake Park Audubon 
(MN) 


1,331 


152 


0.114 


1,616 


87 


0.060 


Grandview/Hickman Mills (MN) 


4,212 


551 


0.131 


7,477 


1,035 


0.138 


Dunkirk City/Jamestown City (NY) 


2,202 


263 


0.119 


5,430 


772 


0.142 


Mean across districts 






0.121 






0.113 



na = not applicable. 

■“Total enrollment for school year 2002-2003 was reported to FNS by each district or the district’s state agency. 

’’Counts of students directly certified are from sampling lists (Table III.4), except that counts from administrative records were 
used for Creve Coeur, Oak Park and River Forest, North Pekin and Marquette and Dunkirk because lists of directly certified 
students were not provided in the sampling process. Data pertain to school year 2002-2003. 

“■Figures for Valley View are for Bollingbrook High School. 

“’Figures for Williamson County are for the schools participating in the pilot project. 
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where: 



(9) Pr(inc< 130) = Pr (Inc< 1 30 & no DC) + Pr(DC) 

= Pr (Inc<130 I no DC)*Pr(no DC) + Pr(DC) 

= Pr (Inc<130 I no DC)*[1 - Pr(DC)] + Pr(DC) . 

In other words, we first estimated the overall probability of students having income below 
130 percent of poverty by using the relationships shown in equation (9) — that is, by summing the 
probability of being directly certified and the joint probability of having an income in this 
category and not being directly certified (which we could estimate from our sample). We then 
estimated our measure of barriers according to equation (8) by using this newly estimated 
probability of being income eligible for free meals along with the probability of being directly 
certified and the conditional probability of being certified free given being income-eligible and 
not directly certified. The latter conditional probability was estimated with sample data using the 
regression described in the previous section. 

We computed the probability of being certified for free or reduced price meals among 
students with household income less than 185 of poverty (measure CB_3b of Table II.2 of 
Volume II) using the same logic: 

(10) Pr(cert FRP I inc<185) = Pr(cert FRP I no DC, inc<185) 

*{ l-[Pr(DC)/Pr(inc<185)] } -i- Pr(DC)/Pr(inc<185) 

where: 

(11) Pr(Inc<185) = Pr (Inc<185 I no DC)*[1 - Pr(DC)] -i- Pr(DC) . 

We used similar procedures to add the directly certified group to our estimates of the simple 
rates of accuracy and targeting efficiency. For the accuracy tables, we calculated the conditional 
probability of being eligible (i.e. having income below the appropriate threshold) given 
certification. For example: 

(12) Pr(inc<130 I cert free) = Pr(inc<130 I cert free, no DC) 

*{ 1 - [Pr(DC)/Pr(cert free)] } -i- Pr(DC)/Pr(cert free) 
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where: 



(13) Pr(cert free) = Pr(cert free I no DC)*[l-Pr(DC)] + Pr(DC) . 

Finally, for the targeting efficiency tables, we calculated the unconditional probability of being 
correctly targeted: 

(14) Pr(Correct) = Pr(Correct I no DC)*[l-Pr(DC)] + Pr(DC) ? 

All of these calculations were performed at the district level. The results presented in the 
tables are the averages across districts, computed separately for Up-Front Documentation and 
Graduated Verification sites and by pilot or comparison status. 

B. ESTIMATING IMPACTS ON ACCURACY AND TARGETING EFFICIENCY 

Accuracy and targeting efficiency are outcomes that are influenced by levels of certification 
among both eligible and ineligible students. By contrast, deterrence focused entirely on those 
ineligible for free/reduced-price meal benefits while barriers focused entirely on those eligible 
for benefits. Thus, accuracy and targeting efficiency are, in effect, summary measures of 
deterrence and barriers, with the impacts of the pilots on both accuracy and targeting efficiency 
influenced by their impacts on deterrence and barriers. 

I. Accuracy 

The accuracy rate is defined as the proportion of certified students who are eligible for the 
level of benefits they are receiving. We examined one measure of free certification accuracy 
(CA_la) and three measures of free and reduced-price certification accuracy (CA_2a, CA_3a, 
and CA_4a) in the descriptive analysis presented in Volume I (defined in Table II.2 in Volume 

^Again, we relied on the assumption that all directly certified students are correctly targeted. 
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I). Our analysis of impacts focuses on the measure of free certification accuracy (CA_la) and 
one of the measures of free and reduced price certification accuracy (CA_4a). 

Because of the relationship between deterrence/barriers and accuracy, it was possible to use 
the results from the models of deterrence and barriers (described in Section A) to determine the 
impacts of the pilots on the accuracy rate. Among non-directly certified students, for example, 

o 

the estimated impact of the pilot on free meal accuracy (measure CA_la) can be written: 

(15) Imp{ Accuracy I CFJ = Pr{/,. < 130 \CF.=\,P^=\]- Pr {/,. < 130 \CF^=\,P^= 0} 

The challenge in calculating this impact is in calculating the terms on the right hand side of 
equation (15) — the probabilities of being eligible for free meals conditional on being certified for 
free meals. We cannot use regression analysis to estimate these conditional probabilities because 
of technical issues concerning the effects of the explanatory variables including pilot status on 
the composition of the estimation sample. In particular, in the case of the probability of being 
certified conditional on being ineligible for benefits (as represented in equations 3 and 4), we 
simply restricted the sample to ineligible students and then regressed certification status on pilot 
status and a variety of other student characteristics. If we were to pursue the analogous 
methodology here, we would restrict the sample to certified students and regress income 
eligibility status (that is, a binary variable indicating whether the students income is no more than 
130 percent of poverty) on pilot status and other student characteristics. The problem with a 
regression model such as this, however, is that the sample in the model is endogenous in that 
pilot status (and other student characteristics) would affect the sample upon which the regression 
was based rather than affecting the dependent variable of the model directly. 

^The logic shown in equation (15) and subsequent equations is for eligibility and 
certification for free meals only. However, we applied the same logic and developed an 
analogous set of equations for our analysis of free or reduced-price eligibility and certification. 
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However, we can use basic probability theory to express the conditional probability of 
eligibility given certification in terms that we have already determined from the deterrence and 
barriers models described above. To illustrate this, the first term on the right hand side of 
equation 15 can be written: 



Pr{l <130|CF =1,P = 1} = 



Pr{l <130&CF =i|P =1} 
Pr{CF =l|P = l} 



(16) 



Pr{CF =l| I <130,/^ =l}Pr{l, <130 I/’ =1} 
Pr{CF =1&I.<130\P. =l}+Pr{CF = 1 & /_ > 130 I T’ = l} 



Pr{CF =l| I_ <130, P. =l}Pr{l, ^BOI/* =1} 

Pr{CF =l| I_ <130,/^ =l}Pr{L < 130 I T’ = 1}+ Pr{CF =l| I, >130,P. =l}Pr{l. >1301/’ =1} 



Each of the probabilities on the right-hand side of this equation are either known or can be 
easily determined. In particular, the conditional probabilities of certification given eligibility and 
given ineligibility will have already been determined in estimating the impacts of the pilots on 
deterrence and barriers. The unconditional probability of eligibility for free meals (I<=130), as 
well as its complement, the unconditional probability of ineligibility for free meals (I>130), can 
be easily determined by calculating the proportion of non-directly certified students in pilot 
districts with incomes below (or above) 130 percent of the FPL. Similarly, the second term on 
the right hand side of equation (15) can be written: 

Pr{l <130 |cf =i,p = o} = 

(17) 

Pr{CF =l| I <130,/’ =0}Pr{L <130 I/’ =0} 

Pr{CF =l| I <130, P. =0}Pr{l ^BOI/’ =0}-l-Pr{CF =l| I > 130, Z’ = 0}Pr{l >1301/’ =0} 
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To summarize, we used the following steps to calculate the impact of the pilot on accuracy: 



1 . Estimate the impact of the pilot on deterrence by estimating equation 1 based on 
a sample of students ineligible for benefits and with a binary variable indicating 
whether the student is certified as the dependent variable. 

2. Based on the results of this estimation, calculate the predicted probability of 
certification conditional on being ineligible among students in pilot districts ( 
Pr{CF=l I I>130, P=l} ) and comparison districts ( Pr{CF=l I I>130, P=0} ). 
These terms are also referred to as the regression-adjusted mean certification 
rates among ineligible students in pilot and comparison districts. 

3. Estimate the impact of the pilot on barriers. 

4. Based on these results, calculate the predicted probability of certification 
conditional on being eligible in pilot ( Pr{CF=l I I<=130, P=l} ) and 
comparison districts ( Pr{CF=l 1 1<=130, P=0} ). 

5. Calculate the predicted probability of the student being eligible and ineligible for 
benefits in both pilot and comparison districts. The predicted probabilities of 
free meal eligibility/ineligibility in pilot districts, for example, would be 
Pr{I<130IP=l} andPr{I>=130IP=l}. 

6. Use all of these conditional predicted probabilities as shown in equation 16 to 
calculate the probability of being income eligible conditional on being certified 
in pilot districts. Do a similar calculation to determine the probability of being 
income eligible conditional on being certified in comparison districts. 

7. Calculate the impact of the pilot as the difference between the probability of 
being eligible conditional on being certified in pilot districts versus comparison 
districts, as shown in equation 15. 



To calculate the standard error of this estimate, we used bootstrap methods. In particular, 
we generated 1,000 replicate samples by sampling from our existing sample with replacement. 
We then estimated the impact of the pilot on accuracy for each replicate sample using the 
methodology described above. The standard deviation of these 1,000 different estimates was our 
estimate of the standard error of the estimated impact of the pilot on accuracy. 



2. Targeting Efficiency 

The rate of targeting efficiency was defined as the proportion of all students in a given 
district whose free or free/reduced-price eligibility status was consistent with their certification 
status. In other words, it was the proportion who were either both certified and eligible for 
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benefits or not certified and not eligible for benefits. We defined three measures of targeting 
efficiency that correspond to the three measures of free and reduced price certification accuracy. 
Our analysis of impacts on targeting efficiency focuses on the broadest of these three measures- 
the percentage of students who are in either of the following two categories: (1) have income 
below 185 percent of the federal poverty level and are certified for free or reduced price meals, 
or (2) have income above 185 percent of the federal poverty level and are not certified for free or 
reduced price meals. (This is measure CTE_3a in Table IV. 8 and Table IV. 16 in Volume I.) 

Like the accuracy rate, the rate of targeting efficiency depends upon the certification status 
among both ineligible students and eligible students. Thus, the impact of the pilot on targeting 
efficiency was determined by the pilots’ impacts on deterrence and accuracy (along with the 
proportion of students who were income eligible and ineligible). The methodology for 
calculating this impact was similar to the methodology for estimating the impact of the pilot on 
accuracy. 

Since the rate of targeting efficiency is the probability of being either certified and 
eligible or not certified and not eligible, the targeting efficiency measures in pilot districts can be 
written in terms that have already been obtained^: 

Pr{ TE.=\\ P; = 1} = Vx{CFRP. = 1 & /,. < 185 1 Z’ = 1}+ Px{CFRP. = 0 & /,. > 185 1 Z’ = 1} 
(18) = Px{CFRP^ = 1 1 Z,. < 185} Pr{Z,. < 185} + Px{CFRP^ = 0 1 Z,. > 185} Pr{Z,. > 185} 

= Px{CFRP. = II Z,. < 185}Pr{Z,. < 185} + [l-Pr{CT’Z?Z} = II Z,. > 185}Pr{Z,. > 185}] 



^In the last two lines of equation 18 the conditional term P=1 has been dropped for 
simplicity. 
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(All terms are as previously defined, exeept that CFRPi=l indieates that a student is eertified 
either for free or for reduced-price meal benefits.) 

A similar equation can be written to obtain the estimated rate of targeting efficiency in 
comparison districts. The difference between these two terms is the estimated impact of the pilot 
on targeting efficiency. 
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X. SUPPLEMENTARY TABULATIONS 



This chapter presents five sets of tabulations that supplement the findings presented in 
Volume I. Section A presents analysis of the changes over time that we observed in the pilot and 
comparison sites, includes supplementary tabulations. Section B presents sample tabulations of 
characteristics by NSLP certification status. Section C presents selected sample characteristics 
for each pilot and comparison district pair. Section D presents key outcomes by district for Up- 
Front Documentation pilot and comparison districts. Section E presents key outcomes by district 
for the Graduated Verification pilot and comparison districts. The tables in Sections B and C 
correspond to characteristics discussed in Chapter III of Volume I, and the tables in Sections D 
and E correspond to outcomes presented in Chapter IV of Volume I. 

A. CHANGES IN PILOT AND COMPARISON SITES AFTER THE PRE-PILOT 

PERIOD 

Chapter III of Volume I presented data that indicated the pilot and comparison sites were 
reasonably well matched on baseline characteristics. However, a concern remains that changes 
over time may have caused the pilot and comparison sites to become less similar than they were 
at baseline. To check on this possibility, we examined changes in district characteristics between 
the pre-pilot period and the pilot period using data covering more recent years from the same 
data sources as were used in selecting the comparison districts.^ Our examination focused on 
factors not likely to have been affected by the demonstration, although we also show changes in 

'Data sources used in this section are 2002 enrollment and NSLP certification and 
participation data from ENS Minimum Data Set, 2001 race/ethnicity data from the NCES CCD, 
and 1999 poverty data from the U.S. Census CPS. Although most of these data are during the 
pilot period, our measure of poverty is actually a pre-pilot measure but is two years more recent 
than the measure we used in selecting comparison sites. 



159 




certification rates, which may have been influenced by the pilot. Table X. 1 shows the changes 
over time in selected characteristics of the Up-Front Documentation and Graduated Verification 
pilot and comparison districts. The discussion in this section focuses on characteristics of the 
districts during the pilot period, based on the most recent data currently available. 

Some characteristics of Up-Front Documentation pilot districts diverged from those of their 
comparison districts over time. The numbers of students enrolled grew in both pilot and 
comparison sites, but the growth was greater in pilot sites, which had fewer students than 
comparison districts in 1997 but surpassed comparison enrollments by 1999. The NSLP free and 
reduced price certification rates in Up-Front Documentation pilots fell (from an average of 23 
percent to 19 percent) between 1999 and 2001, while the certification rates in comparison sites 

'•y 

rose slightly (from 22 percent to 24 percent). The racial/ethnic distributions of pilot and 
comparison sites as measured during the pilot period still matched closely. 

In the Graduated Verification demonstrations, differences in characteristics between pilot 
and comparison districts changed only slightly between the pre-pilot and pilot periods. 
Enrollments in pilot districts fell, while comparison enrollments rose slightly, resulting in a 
somewhat larger size difference. The percentage of white students fell slightly in both pilot 
districts and comparison districts, but the change was greater in pilots, resulting in slightly less 
similar race/ethnicity distributions. 

In summary, while the pilot and comparison districts developed differently over time, and 
were less similar after baseline than they were at baseline, within each pilot alternative the pilot 
and comparison sites remain well-matched. 



^This could be a result of changes in poverty rates. However, it could also have been 
affected by the demonstration, so this change should be interpreted with caution. 
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TABLE X.l 



CHARACTERISTICS OE UP-ERONT DOCUMENTATION AND GRADUATED VERIEICATION 
PILOT AND COMPARISON DISTRICTS 



Characteristics 






Pre-Pilot* 






During Pilot'’ 




Pilot 

Districts 


Comparison 

Districts 


Difference 


Pilot 

Districts 


Comparison 

Districts 


Difference 


Up-Front Documentation 
















District Size 


1999- 














Number of Students (Mean)*^ 


2002 


3,381.0 


3,446.2 


-65.2 


3,815.6 


3,473.7 


341.9 


NSLP Certification Status 


1999- 














(Percentage) 


2002 














Free 




16.5 


16.5 


0.0 


14.7 


18.1 


-3.4 


Reduced Price 




5.9 


5.9 


0.0 


5.1 


6.3 


-1.2 


Poverty Rate Among School- 


1997- 














Age Children 


1999 


10.4 


10.3 


0.1 


9.3 


9.9 


-0.6 


Race/Ethnicity (Percentages) 


1999- 














White, Non-Hispanic 


2001 


88.4 


88.7 


-0.3 


87.2 


87.6 


-0.4 


Black, Non-Hispanic 




6.9 


6.9 


0.0 


7.4 


7.2 


0.2 


Hispanic 




3.4 


3.0 


0.4 


4.1 


3.8 


0.3 


Native American 




0.2 


0.0 


0.2 


0.2 


0.1 


0.1 


Asian, Pacific Islander 




1.1 


1.3 


-0.2 


1.1 


1.3 


-0.2 



Graduated Verification 

District Size 

Number of Students (Mean)*^ 


1999- 

2002 


2,587.0 


4,537.5 


-1,950.5 


2,581.7 


4,571.7 


-1,990.0 


NSLP Certification Status 
(Percentage) 

Free 


1999- 

2002 


34.9 


32.2 


2.7 


34.3 


37.3 


-3.0 


Reduced Price 




9.6 


9.6 


0.0 


7.8 


9.3 


-1.5 


Percentage of School-Age 
Children Below 100 Percent of 
the Federal Poverty Level*^ 


1997- 

1999 


19.4 


18.8 


0.6 


22.4 


16.4 


6.0 


Race/Ethnicity (Percentages) 
White, Non-Hispanic 


1999- 

2001 


65.8 


70.6 


-4.8 


62.6 


68.7 


-6.1 


Black, Non-Hispanic 




18.3 


24.0 


-5.7 


20.2 


24.8 


-4.6 


Hispanic 




14.1 


4.0 


10.1 


15.2 


4.9 


10.3 


Native American 




1.3 


0.9 


0.4 


1.4 


0.8 


0.6 


Asian, Pacific Islander 




0.5 


0.6 


-0.1 


0.7 


0.8 


-0.1 



“Source: 1999 FNS Minimum Data Set; 1999 NCES Common Core of Data; 1997 Census CPS. 
'’Source; 2002 FNS Minimum Data Set; 2001 NCES Common Core of Data; 1999 Census CPS. 
‘Includes students enrolled in schools implementing the pilot. 

‘'Change is from one pre-pilot period to later pre-pilot period. 
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B. SAMPLE CHARACTERISTICS BY NSLP CERTIFICATION STATUS 



This section presents sample characteristics by NSLP certification status. Table X.2 
displays this data for Up-Front Documentation pilot and comparison districts, and Table X.3 
shows this data for Graduated Verification districts. Households certified to receive free or 
reduced price meals are more likely to be single-parent households and include respondents with 
lower levels of education and employment than non-certified households. Certified households 
have lower incomes, higher rates of public assistance receipt, and higher pre -pilot certification 
rates. 

C. SELECTED SAMPLE CHARACTERISTICS BY DISTRICT 

This section presents selected sample household characteristics individually for each pilot 
and comparison district pair.^ Table X.4 displays survey data on household size and structure. 
Table X.5 addresses parent’s education. Table X.6 shows household income, and Table X.7 
presents race/ethnicity data. Table X.8 shows survey data on pre-pilot certification status, for 
those students old enough to have been in school before the pilot began. In all these tables, 
households are weighted to adjust for non-response; directly certified students are not included. 

There were a few notable differences across sites. Up-Front Documentation Pilot District #9 
had very few single-parent households (7.8, compared to 23.5 on average across pilot sites). Up- 
Front Documentation District Pair #7 included almost no respondents with a college degree (4.8 
percent in the pilot and 3.3 percent in the comparison), while the majority of Up-Front 



The tables present the same set of sample characteristics discussed in Volume I, Chapter 3. 
Each district is weighted equally so the statistics presented are representative of the average 
district. 

In this set of tables, households are weighted to adjust for non-response, so the statistics 
presented are representative of the survey sample, which excluded directly certified students. 
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TABLE X.2 



CHARACTERISTICS OE SAMPLE MEMBERS IN UP-FRONT DOCUMENTATION PILOT 
AND COMPARISON DISTRICTS, BY NSLP CERTIFICATION STATUS 



Certified Free Certified Reduced-Price Not Certified 





Pilot 


Comparison 


Pilot 


Comparison 


Pilot 


Comparison 


Characteristics 


Districts 


Districts 


Districts 


Districts 


Districts 


Districts 


Household Size 














Number of household members (Mean) 


4.3 


4.6 


4.6 


4.5 


4.2 


4.3 


Number of children in household (Mean) 


2.7 


2.8 


2.7 


2.7 


2.1 


2.2 


Household Structure (Percentages) 














Two-parent household 


48.2 


54.3 


71.2 


63.4 


79.7 


83.0 


Single-parent household 


50.1 


43.1 


26.2 


35.3 


19.4 


15.9 


Other household structure 


1.8 


2.7 


2.6 


1.4 


1.0 


1.1 


Survey Respondent’s Educational Attainment 
(Percentages) 














Lacks a high school diploma 


20.0 


21.5 


8.7 


12.7 


4.8 


7.8 


High school diploma only 

Some postsecondary education but lacks a 


45.9 


51.4 


46.7 


52.2 


40.7 


38.8 


college degree 


27.4 


22.6 


39.0 


30.1 


27.0 


31.9 


College degree or more 


6.7 


4.6 


5.6 


5.0 


27.5 


21.5 


Employment Status Of Household Members 














Survey respondent is employed (Percentage) 
Number of employed adults in household 


52.3 


58.2 


54.5 


70.3 


74.1 


77.6 


(Mean) 


0.8 


1.1 


1.2 


1.3 


1.6 


1.7 


Household Income (Percentages) 














Less than 130 percent of FPL“ 


79.6 


77.5 


27.4 


32.0 


10.8 


9.5 


131 to 185 percent of FPL 


10.3 


12.2 


44.9 


45.7 


10.8 


11.4 


186 to 400 percent of FPL 


8.1 


10.0 


25.0 


21.3 


40.9 


49.1 


More than 400 percent of FPL 


2.1 


0.3 


2.8 


0.9 


37.5 


30.0 


Percentage Receiving Public Assistance 














Percentage receiving food stamp benefits 


36.0 


32.4 


2.5 


3.1 


2.7 


1.8 


Percentage receiving TANF 


4.5 


6.1 


1.1 


0.8 


0.5 


0.7 


Percentage receiving other benefits 


25.1 


18.6 


11.2 


7.7 


10.1 


6.7 


Percentage Residing in Public Housing or 














Receiving Housing Subsidy 


15.5 


11.1 


3.0 


5.7 


0.9 


0.6 


Percentage Who Own Their Home 


37.7 


43.1 


62.0 


65.7 


84.0 


82.4 


Vehicle Ownership (Percentage) 


65.0 


73.4 


84.1 


83.9 


88.6 


91.1 


Number of Vehicles Owned By All Household 














Members (Mean) 


1.0 


1.2 


1.4 


1.5 


1.9 


2.0 


Household Mobility 














Number of Times Respondent Has Moved 
During Past Two Years (Means) 


0.5 


0.5 


0.3 


0.2 


0.2 


0.2 


Has Moved And Changed School Districts 
During Past Two Years (Percentage) 


21.2 


16.8 


9.4 


12.0 


8.6 


7.4 
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TABLE X.2 (continued) 





Certified Free 


Certified Reduced-Price 


Not Certified 


Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Race/Ethnicity (Percentages) 


White, Non-Hispanic 


71.8 


78.1 


77.9 


74.3 


84.5 


85.7 


Black, Non-Hispanic 


17.0 


8.8 


9.7 


12.4 


6.4 


5.4 


Hispanic 


4.0 


6.8 


6.1 


3.9 


1.5 


2.1 


Native American 


0.7 


0.0 


0.0 


1.0 


0.7 


0.2 


Asian, Pacific Islander 


0.3 


1.2 


0.4 


1.1 


0.4 


1.3 


Other 


1.7 


1.7 


1.0 


3.6 


2.7 


2.2 


Mixed race 


4.5 


3.4 


4.9 


3.7 


3.9 


3.0 


English Primary Language Spoken at Home 


(Percentage) 


96.9 


93.7 


94.4 


96.6 


97.5 


97.7 


Grade Level of Child (Percentages) 


Grade 9 to 12 


29.4 


22.6 


23.5 


32.3 


39.4 


36.7 


Grade 6 to 8 


19.6 


27.0 


27.1 


20.2 


20.2 


24.6 


Grade 3 to 5 


32.4 


27.2 


27.4 


29.7 


19.5 


21.5 


Grade 1 to 2 


15.5 


18.8 


18.8 


17.0 


13.6 


10.7 


Kindergarten or Pre-K 


3.2 


4.3 


2.6 


0.8 


6.7 


6.4 


Pre-Pilot Free or Reduced-Price Certification 


Status (Percentage) 


77.8 


75.0 


64.0 


53.7 


13.5 


10.4 


Sample Size 


204 


222 


115 


154 


631 


612 



■“The lowest income category (less than 130% of FPL) includes families categorically eligible for free meals (based on receipt of TANF or 
food stamps or foster child status), regardless of their actual income. 

FPL = federal poverty level. 
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TABLE X.3 



CHARACTERISTICS OF SAMPLE MEMBERS IN GRADUATED VERIFICATION PILOT 
AND COMPARISON DISTRICTS, BY NSLP CERTIFICATION STATUS 



Certified Free Certified Reduced-Price Not Certified 





Pilot 


Comparison 


Pilot 


Comparison 


Pilot 


Comparison 


Characteristics 


Districts 


Districts 


Districts 


Districts 


Districts 


Districts 


Household Size 














Number of household members (Mean) 


4.4 


4.9 


4.5 


4.4 


4.2 


4.4 


Number of children in household (Mean) 


2.7 


3.2 


2.6 


2.5 


2.2 


2.2 


Household Structure (Percentages) 














Two-parent household 


49.1 


52.2 


65.6 


51.6 


72.7 


82.6 


Single-parent household 


48.8 


44.7 


33.3 


43.3 


26.8 


15.5 


Other household structure 


2.1 


3.2 


1.1 


5.1 


0.5 


1.9 


Survey Respondent’s Educational Attainment 
(Percentages) 














Lacks a high school diploma 


25.6 


23.9 


23.1 


7.7 


7.1 


6.4 


High school diploma only 

Some postsecondary education but lacks a 


35.2 


39.1 


24.2 


28.7 


28.9 


31.9 


college degree 


31.2 


34.2 


38.7 


56.3 


41.8 


42.2 


College degree or more 


8.1 


2.8 


14.1 


7.4 


22.1 


19.5 


Employment Status Of Household Members 














Survey respondent is employed (Percentage) 
Number of employed adults in household 


61.8 


51.4 


69.8 


78.2 


83.8 


82.5 


(Mean) 


1.0 


0.9 


1.3 


1.3 


1.7 


1.7 


Household Income (Percentages) 














Less than 130 percent of FPL“ 


71.6 


73.0 


22.7 


24.9 


14.8 


11.1 


131 to 185 percent of FPL 


22.4 


19.4 


61.0 


41.3 


11.2 


9.0 


186 to 400 percent of FPL 


4.6 


7.2 


14.6 


32.9 


48.0 


53.6 


More than 400 percent of FPL 


1.4 


0.4 


1.7 


0.9 


26.0 


26.3 


Percentage Receiving Public Assistance 














Percentage receiving food stamp benefits 


37.6 


33.3 


5.2 


2.1 


5.3 


5.5 


Percentage receiving TANF 


16.0 


15.9 


3.0 


0.0 


1.6 


2.5 


Percentage receiving other benefits 


16.8 


22.0 


15.6 


3.1 


9.2 


9.6 


Percentage Residing in Public Housing or 














Receiving Housing Subsidy 


23.3 


19.0 


4.1 


1.5 


2.5 


1.6 


Percentage Who Own Their Home 


41.6 


46.3 


52.9 


60.8 


75.9 


77.2 


Vehicle Ownership (Percentage) 


65.8 


73.4 


77.9 


94.3 


88.6 


87.5 


Number of Vehicles Owned By All Household 














Members (Mean) 


1.0 


1.0 


1.3 


1.4 


1.8 


1.9 


Household Mobility 














Number of Times Respondent Has Moved 
During Past Two Years (Means) 


0.6 


0.5 


0.4 


0.3 


0.2 


0.2 


Has Moved And Changed School Districts 
During Past Two Years (Percentage) 


20.4 


19.9 


10.0 


11.2 


7.1 


3.5 
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TABLE X.3 (continued) 





Certified Free 


Certified Reduced-Price 


Not Certified 


Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Race/Ethnicity (Percentages) 


White, Non-Hispanic 


45.3 


51.7 


65.3 


58.0 


71.4 


61.2 


Black, Non-Hispanic 


19.5 


25.2 


9.9 


21.0 


18.6 


24.0 


Hispanic'’ 


23.5 


13.6 


16.7 


7.4 


5.7 


6.3 


Native American 


1.8 


1.0 


0.0 


0.0 


0.4 


0.0 


Asian, Pacific Islander 


0.9 


0.0 


0.0 


0.0 


0.0 


0.9 


Other 


0.3 


1.0 


0.0 


1.3 


0.5 


2.8 


Mixed race 


8.8 


7.6 


8.0 


12.3 


3.4 


4.9 


English Primary Language Spoken at Home 


(Percentage) 


82.8 


84.3 


91.5 


92.9 


96.7 


93.3 


Grade Level of Child (Percentages) 


Grade 9 to 12 


15.7 


15.1 


18.7 


15.5 


36.0 


34.9 


Grade 6 to 8 


28.9 


28.2 


23.7 


28.0 


27.9 


21.5 


Grade 3 to 5 


30.4 


28.2 


39.5 


28.0 


17.8 


17.0 


Grade 1 to 2 


20.4 


14.3 


9.2 


8.9 


11.0 


19.1 


Kindergarten or Pre-K 


4.7 


14.2 


8.9 


19.0 


7.0 


7.5 


Pre-Pilot Free or Reduced-Price Certification 


Status (Percentage) 


73.2 


78.4 


74.9 


80.9 


24.1 


19.8 


Sample Size 


221 


229 


66 


85 


243 


221 



■“The lowest income category (less than 130% of FPL) includes families categorically eligible for free meals (based on receipt of TANF or 
food stamps or foster child status), regardless of their actual income. 

'’Hispanic students were oversampled in Jamestown City to make the comparison sample more similar to the Dunkirk City pilot sample in 
terms of the percentage of students who were Hispanic. 

FPL = federal poverty level. 
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PARENT’S EDUCATION, BY PILOT AND COMPARISON DISTRICT PAIR 
(Percentage of Sample) 
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HOUSEHOLD INCOME AS A PERCENTAGE OE EEDERAL POVERTY LEVEL, BY PILOT AND COMPARISON DISTRICT PAIR 

(Percentage of Households) 
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RACE/ETHNICITY, BY PILOT AND COMPARISON DISTRICT PAIR 
(Percentage of Students) 
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TABLE X.8 



PRE-PILOT NSLP CERTIEICATION STATUS OP STUDENTS NOT DIRECTLY CERTIPIED, 
BY PILOT AND COMPARISON DISTRICT PAIR 
(Percentage of Students) 





Certified Tree or Reduced-Price 


Not Certified 






Comparison 




Comparison 


Site Pair“ 


Pilot District 


District 


Pilot District 


District 


Up-Front Documentation 










District Pair 1 


38.9 


18.7 


61.1 


81.3 


District Pair 2 


26.1 


16.2 


73.9 


83.8 


District Pair 3 


14.1 


23.3 


85.9 


76.7 


District Pair 4 


22.6 


14.3 


77.4 


85.7 


District Pair 5 


15.8 


32.6 


84.2 


67.4 


District Pair 6 


33.6 


28.2 


66.4 


71.8 


District Pair 7 


41.0 


25.1 


59.1 


74.9 


District Pair 8 


18.4 


26.8 


81.6 


73.2 


District Pair 9 


9.9 


9.3 


90.1 


90.7 


Overall for Up-Front 
Documentation Districts 


24.5 


21.6 


75.5 


78.4 



Graduated Verification 










District Pair 1 


23.7 


28.0 


76.3 


72.1 


District Pair 2 


35.0 


49.3 


65.0 


50.7 


District Pair 3 


55.9 


40.8 


44.1 


59.2 


Overall for Graduated 
Verification Districts 


38.2 


39.3 


61.8 


60.7 



^able uses a different district designation than tables in Chapter IX and Tables X.9-X.16. 
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Documentation Pilot District #8 respondents (69.8 percent) had a degree. Up-Front 
Documentation Pilot District #8 households also had relatively high incomes (63.3 percent have 
incomes above 400% of poverty, compared to 32.5 on average across pilot sites). Graduated 
Verification District Pair #2 had high proportions of African-American families (50.7 in the pilot 
and 68.9 percent in the comparison), as did Up-Front Documentation District Pair #8 (24.2 in the 
pilot and 31.9 percent in the comparison). Graduated Verification District Pair #3 had high 
proportions of Hispanic families (26.8 in the pilot and 21.8 percent in the comparison). Up-Front 
Documentation District Pair #9 had low pre-pilot NSLP certification rates (9.9 percent in the 
pilot and 9.3 percent in the comparison), and Graduated Verification Pilot District #3 was the 
only site with a certification rate above 50 percent. 

D. PRIMARY OUTCOMES FOR UP-FRONT DOCUMENTATION DISTRICTS, BY 
DISTRICT 

This section presents tabulations of each main outcome reported in Chapter IV of Volume 1 
for each pilot and comparison district in the Up-Front Documentation evaluation. Tables X.9- 
X.12 present mean outcomes by district, weighted to adjust for non-response.^ Sample sizes by 
district are too small to draw valid conclusions about the district. Consequently, we have not 
presented standard errors of the estimates. The data are presented to allow the reader of Volume 
I to assess the variability of the findings at the district level. 

E. PRIMARY OUTCOMES FOR GRADUATED VERIFICTION DISTRICTS, BY 
DISTRICT 

This section presents tabulations of the main outcomes for each pilot and comparison district 
in the Graduated Verification Pilot demonstration. As with the corresponding estimates for the 



^Thus the statistics presented are representative of the survey sample, which excluded 
directly certified students. 
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TABLE X.9 



RATES OE CERTIEICATION AMONG STUDENTS INELIGIBLE EOR EREE OR REDUCED-PRICE MEALS 
IN UP-ERONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS 



Site Pair“ 


Free Certification Rate Among 
Students with Income >130% 
FPL (CD_1)'’ 


Free or Reduced-Price 
Certification Rate Among 
Students with Income >185% 
FPL (CD_2)'’ 


Pilot District 


Comparison 

District 


Pilot District 


Comparison 

District 


UFD District Pair 1 


1.1 


5.3 


1.3 


6.4 


UFD District Pair 2 


12.7 


4.2 


8.5 


3.1 


UFD District Pair 3 


5.4 


3.5 


7.6 


1.7 


UFD District Pair 4 


2.0 


1.5 


2.7 


1.9 


UFD District Pair 5 


2.0 


1.1 


2.2 


0.5 


UFD District Pair 6 


5.6 


4.2 


7.9 


2.8 


UFD District Pair 7 


0.6 


6.8 


0.7 


7.7 


UFD District Pair 8 


0.7 


4.7 


1.0 


6.3 


UFD District Pair 9 


1.0 


4.1 


1.9 


5.6 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X. 10 



RATES OE CERTIEICATION AMONG NON-DIRECTLY CERTIEIED STUDENTS 
ELIGIBLE EOR EREE OR REDUCED-PRICE MEALS 
IN UP-ERONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS 



Site Pair'* 


Free Certification Rate Among 
Students with Income <130% 
FPL and Not Directly Certified 
(CB_la)'’ 


Free or Reduced-Price 
Certification Rate Among 
Students with Income <185% 
FPL and Not Directly Certified 
(CB_3a)'’ 


Pilot 

District 


Comparison 

District 


Pilot 

District 


Comparison 

District 


UFD District Pair 1 


39.6 


71.2 


28.8 


65.0 


UFD District Pair 2 


70.2 


55.4 


63.5 


52.5 


UFD District Pair 3 


27.9 


42.0 


42.4 


41.7 


UFD District Pair 4 


38.7 


62.8 


37.5 


51.0 


UFD District Pair 5 


69.4 


50.9 


65.4 


43.8 


UFD District Pair 6 


68.5 


58.8 


62.8 


52.5 


UFD District Pair 7 


41.7 


58.5 


24.4 


60.5 


UFD District Pair 8 


23.1 


24.1 


19.7 


30.6 


UFD District Pair 9 


46.5 


61.0 


37.4 


56.8 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X. 11 



ACCURACY RATES AMONG STUDENTS CERTIEIED EOR EREE OR REDUCED-PRICE MEALS 
AND NOT DIRECTLY CERTIEIED IN UP-ERONT DOCUMENTATION 
PILOT AND COMPARISON DISTRICTS 



Site Pair“ 


Free Certification 
Accuracy Among Free 
Approved Non-Directly 
Certified Students 
(CA_la)'’ 


Not Over 185% FPL 
Among Free and 
Reduced-Price Approved 
Non-Directly Certified 
Students (CA_4a)'’ 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Estimates Excluding Directly Certified Students 










UFD District Pair 1 


89.9 


82.2 


92.1 


86.7 


UFD District Pair 2 


65.0 


81.8 


86.4 


92.9 


UFD District Pair 3 


58.6 


73.7 


74.2 


92.5 


UFD District Pair 4 


78.4 


88.3 


80.8 


92.4 


UFD District Pair 5 


88.0 


88.2 


91.7 


96.5 


UFD District Pair 6 


83.4 


82.0 


87.3 


92.2 


UFD District Pair 7 


75.7 


66.7 


78.6 


73.6 


UFD District Pair 8 


80.8 


47.3 


84.7 


61.8 


UFD District Pair 9 


96.4 


87.4 


94.9 


89.5 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X.12 



RATES OE TARGETING EEEICIENCY AMONG NON-DIRECTLY CERTIEIED STUDENTS IN 

UP-ERONT DOCUMENTATION 
PILOT AND COMPARISON DISTRICTS 



Site PaiU 



Targeting Efficiency Measures Among Non-Directly 
Certified Students (CTE_3a)*’ 

Pilot Comparison 

Districts Districts 



Estimates Excluding Directly Certified Students 



UFD District Pair 1 


74.6 


82.5 


UFD District Pair 2 


78.6 


77.7 


UFD District Pair 3 


75.4 


79.2 


UFD District Pair 4 


83.4 


83.1 


UFD District Pair 5 


88.9 


86.0 


UFD District Pair 6 


78.6 


79.9 


UFD District Pair 7 


92.4 


83.9 


UFD District Pair 8 


81.7 


78.0 


UFD District Pair 9 


68.5 


77.2 



Note: Efficient targeting is defined as being correctly assigned to either the certified or non-certified group given 

household income. It is calculated as the percentage of students who are either (a) eligible for benefits and 
certified, or (b) not eligible for benefits and not certified. 

■“Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 

'’Definitions of measures are provided in Table II.2 of Volume I. 
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Up-Front Documentation districts, sample sizes are not adequate to support district level analysis 



and standard errors of the estimates are not presented. Tables X.13-X.16 present weighted mean 
outcomes by district. 
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TABLE X. 13 



RATES OE CERTIEICATION AMONG STUDENTS INELIGIBLE EOR EREE OR REDUCED-PRICE MEALS 
IN GRADUATED VERIEICATION PILOT AND COMPARISON DISTRICTS 



Site Pair'* 


Tree Certification Rate Among 
Students with Income >130% 
PPL (CD_1)'’ 


Pree or Reduced-Price 
Certification Rate Among 
Students with Income >185% 
PPL (CD_2)'* 


Pilot 

District 


Comparison 

District 


Pilot 

District 


Comparison 

District 


GV District Pair 1 


8.3 


12.4 


3.9 


13.2 


GV District Pair 2 


7.2 


7.0 


6.7 


10.5 


GV District Pair 3 


6.6 


6.5 


3.8 


5.4 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X. 14 



RATES OE CERTIEICATION AMONG STUDENTS ELIGIBLE EOR EREE OR REDUCED-PRICE MEALS 
IN GRADUATED VERIEICATION PILOT AND COMPARISON DISTRICTS 



Tree Certification Rate Among 
Students with Income <130% 
PPL and Not Directly Certified 
(CB_la)'’ 



Pree or Reduced-Price Certification 
Rate Among Students with Income 
<185% PPL and Not Directly 
Certified (CB_3a)'’ 



Site Pair'* 


Pilot 

District 


Comparison 

District 


Pilot 

District 


Comparison 

District 


GV District Pair 1 


63.5 


71.1 


65.8 


81.8 


GV District Pair 2 


66.0 


65.9 


66.8 


62.3 


GV District Pair 3 


37.8 


70.7 


48.7 


72.6 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X. 15 



ACCURACY RATES AMONG STUDENTS CERTIEIED EOR EREE OR REDUCED-PRICE MEALS 

IN GRADUATED VERIEICATION 
PILOT AND COMPARISON DISTRICTS 



Site Pair“ 


Tree Certification 
Accuracy Among Tree 
Approved Non-Directly 
Certified Students 
(CA_la)'’ 


Not Over 185% PPL 
Among Eree and 
Reduced-Price Approved 
Non-Directly Certified 
Students (CA_4a)'’ 


Pilot 

Districts 


Comparison 

Districts 


Pilot 

Districts 


Comparison 

Districts 


Estimates Excluding Directly Certified Students 










GV District Pair 1 


74.5 


77.7 


92.0 


86.2 


GV District Pair 2 


88.2 


86.0 


94.5 


88.0 


GV District Pair 3 


52.1 


55.5 


85.9 


81.7 



■‘Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 
'’Definitions of measures are provided in Table II.2 of Volume I. 
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TABLE X.16 



RATES OE TARGETING EEEICIENCY AMONG CERTIEIED STUDENTS IN 
GRADUATED VERIEICATION 
PILOT AND COMPARISON DISTRICTS 



Targeting Efficiency Measures Among Non-Directly 
Certified Students (CTE_3a)'’ 


Site Pair“ 


Pilot 

Districts 


Comparison 

Districts 


Estimates Excluding Directly Certified Students 






GV District Pair 1 


83.7 


84.3 


GV District Pair 2 


76.5 


74.5 


GV District Pair 3 


80.9 


89.1 



Note: Efficient targeting is defined as being correctly assigned to either the certified or non-certified group given 

household income. It is calculated as the percentage of students who are either (a) eligible for benefits and 
certified, or (b) not eligible for benefits and not certified. 

■“Common site pair designations are used in all tables in Chapters IX and Tables X.9-X.16. 

'’Definitions of measures are provided in Table II.2 of Volume I. 
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